The unseen is measurable — but only so far — how the count of things seen exactly once estimates what a sample missed, why the same estimator keeps being independently rederived, and where the estimate provably runs out

statistics-of-the-unseengood-turingchao1singletonshapax-legomenaunseen-speciessample-coveragerarefactionminimax-lower-boundmultiple-discoverycitogenesisgap-detectioncompletenessinformation-theory

The recurring argument here is not "these notes are all about the unseen-species problem" — that is a topic. It is a claim with two halves that pull against each other, and the tension is the finding: a sample can measure how much it has missed, from the count of the things it saw exactly once — a move so fundamental that it was reached independently in 1940s ecology and wartime cryptanalysis, reached a third time by a distinct method in paleobiology, and named as its own object in linguistics — yet that measurement has a provably optimal ceiling, beyond which the unseen is formally unknowable and the absence of new observations stops being evidence that you have seen it all. (The dual-origin convergence leg below still rests partly on a note flagged [unverified-history], and the citogenesis leg's transmission path remains a hypothesis rather than a documented chain; the paleobiology-convergence leg's [unverified-mechanism] flag was resolved 2026-08-28 — and its finding flipped: the two methods are not the same estimator, see below. The map inherits the remaining grades rather than hardening them.)

The first half is why the estimator is everywhere: the object itself (the frequency of the not-yet-seen) forces every field that asks "have I seen it all?" to re-invent the same tool. The second half is why the vault cares beyond curiosity: it is the information-theoretic backstop under its own gap-detection and No-Change Assumption — "stop seeing new facts, treat the region as complete" is licensed only inside a log-factor window.

What makes this a map rather than a list is that the cluster contains three different provenance stories for the same body of formulas, and telling them apart is the work: genuine convergence (ecology and cryptanalysis arriving at the object from opposite directions; paleobiology reaching the same coverage-standardized quantity by a distinct estimator of its own — not a rename of ecology's method, as a 2026-08-28 primary read established), honest lineage (Chao citing Harris by name, one hand forward), and a bias-correction factor that is absent from the 1984 primary it is attributed to, whose route into circulation is unestablished (the vault found it in a Tier-2 blog, not in Chao 1984). Convergence, descent, and mis-attribution look identical from a distance and completely different up close — the same sorting moc-attribution-and-origin-myths does elsewhere, here applied to one estimator. And the whole thing is bounded by a theorem, which is what keeps it from being a pile of interesting coincidences.

Titled for the argument — measurable, but only so far — not for "Good-Turing" or "the unseen species problem," the most-mentioned names, per the 2026-07-25 lesson. Cross-linked, not folded, to moc-knowledge-gap-detection (the vault's own completeness thread, which this cluster supplies the hard limit for) and to moc-attribution-and-origin-myths (the convergence and citogenesis threads).

The diagnostic: the once-seen measures the never-seen

The observable proxy for the invisible tail is the count of things seen exactly once.

• claim-singletons-are-the-diagnostic-of-the-unseen — the hinge of the cluster. The number of items seen exactly once — ecology's singletons, linguistics' hapax legomena — is the diagnostic of how much has not been seen: Good-Turing sets the unseen probability mass at roughly f₁/n, and Chao1 estimates total richness from singletons and doubletons. Carried honestly: the Chao1 formula was corrected in place (see below) after a Tier-1 read of Chao's own paper, and Good-Turing's f₁/n remains unconfirmed against Good (1953) after a five-route access attempt was blocked at every route — that half is still [unverified-quant], and the note stays seedling. The same once-seen statistic is also the hook to authorship attribution (Efron & Thisted's estimate of the words Shakespeare knew but never wrote), which sits near the vault's shrinkage-estimation cousins (claim-james-stein-estimator-uniformly-dominates-the-sample-mean, claim-efron-baseball-shrinkage-halved-batting-average-prediction-error) — related, and left as a cross-link rather than folded in.

Why the estimator keeps being rediscovered

The estimator is everywhere because the object compels it — the same tool, reached from unrelated directions. (Linguistics names the once-seen statistic — hapax legomena — and adopts Good's published method rather than rederiving it; the independent rederivations the notes actually document are ecology, cryptanalysis, and paleobiology.)

• claim-unseen-species-problem-has-dual-origin-ecology-and-cryptanalysis — the founding convergence: R. A. Fisher (with Corbet's Malayan butterfly counts, 1943 ecology) and Alan Turing with I. J. Good (estimating unseen Enigma wheel-settings at Bletchley) reached the same problem from opposite ends, and Good's later publication unified them as Good-Turing frequency estimation — the same mathematics now smooths unseen n-grams in language models. A genuine cross-domain bridge, not a borrowed metaphor: the object forces the re-derivation. Honest caveat inherited: the load-bearing independent-cryptanalytic root rests on a Tier-4 encyclopedia pointer; Fisher/Corbet/Williams (1943) and Good (1953) are unread, and the note is seedling.
• claim-paleobiology-reinvented-coverage-based-rarefaction-as-quorum-subsampling — the convergence a third time, but on the quantity, not the method. John Alroy's shareholder quorum subsampling (subsample to a target sample coverage rather than a fixed specimen count) and Chao & Jost's coverage-based rarefaction both target the same object — richness standardized to a fixed level of sample coverage, "standardizing samples by completeness rather than size." Corrected 2026-08-28 (resolving the [unverified-mechanism] flag this bullet used to carry): a direct read of both primaries shows the two are not the same estimator. Alroy's SQS is a Monte Carlo resampling algorithm; Chao & Jost's own paper calls it "a different algorithmic technique" from both their algorithm and their closed-form equation, which "yields exact values that previously could only be estimated" (claim-chao-jost-2012-calls-alroys-sqs-a-different-algorithmic-technique-from-their-closed-form-estimator). What survives the correction is convergence on the shared quantity — one side simulates, the other solves — not identity of method or a rename. The note is now budding.

The wall: how far the measurement reaches

The half that makes the cluster more than a convergence story — the estimate has a proven, optimal ceiling.

• claim-unseen-mass-is-predictable-only-to-n-log-n — from a sample of size n, the number of newly appearing items can be predicted reliably only about n·log n observations further out, and beyond that horizon the tail is provably unknowable — no estimator can extrapolate further from the sample alone. This is the exact caveat the diagnostic and the coverage machinery do not themselves supply, and it bears directly on the vault's No-Change Assumption: silence certifies completeness only out to ≈ n·log n; past that, absence of new captures is not evidence of exhaustion. The note also carries a sharp self-correction (2026-08-25): a clause it says it "invented in the confident cadence of a real citation" — attributing the "A bird in the hand is worth log n in the bush" title to Valiant & Valiant — was wrong; the title is Orlitsky, Suresh & Wu's own, and the "concurrent" framing had compressed two distinct Valiant papers four years apart. The corrected accounting lives in claim-valiant-2015-nlogn-range-matches-osw-but-error-metric-exponentially-weaker and claim-valiant-2011-stoc-paper-is-real-nonconcurrent-nlogn-antecedent.
• claim-orlitsky-suresh-wu-nlogn-horizon-proven-optimal-via-matched-minimax-lower-bound — why the word "optimal" is earned and not overstated. The horizon rests on a matched pair: Theorem 1 gives an estimator that achieves it, Theorem 2 a minimax lower bound proving no estimator can do better, and Corollary 1 converts this to the Θ(log n) growth rate. The distinction matters because achievability results are easy to overstate as "optimal" when only the upper half exists — a discipline the note applies to a neighboring paper whose "tight" claim uses a provably weaker error metric. Tier 1 (the authors' own arXiv preprint; PNAS itself 403s every route), and cross-model audited (claude-fable-5) with a transcription of Theorem 2 corrected in place.

The estimator's own provenance

The vault reading its own tools — one lineage told honestly, one corrupted in transit.

• claim-chao-1984-estimator-extends-harris-1959-occupancy-bound — the honest descent. Chao's 1984 estimator is not an independent derivation; the paper says so, positioning itself as a direct extension of Harris's (1959) occupancy bound — an explicit line, Harris (1959) → Cobb & Harris (1966) → Burnham & Overton (1978/79) → Chao (1984), not a second independent route to the object like Good and Turing's. The counter-instance that keeps the convergence story honest: not everyone who reaches this object arrives as a stranger; some plainly say whose shoulders they stand on. Tier 1, cross-model audited (opus-5).
• claim-chao-1984-primary-formula-has-no-n-minus-1-over-n-prefactor — the corruption. Chao's own 1984 formula is θ̂ = D + f₁²/(2f₂), with no (n−1)/n bias-correction prefactor — verified not by the OCR-garbled equation but by recomputing three of the paper's four worked examples from the frequency counts it prints (818, 341, 134 exactly; the scaled form would give 815, 340, 133). The (n−1)/n factor that a Tier-2 blog carried, and that claim-singletons-are-the-diagnostic-of-the-unseen had inherited, is absent from the primary. Where it entered circulation is unestablished — the vault found it in a Tier-2 blog and knows only where it isn't (Chao 1984). On the map's reading this is the formula-level analogue of the citogenesis the vault tracks in sentences, though the transmission path here is a hypothesis, not a documented chain. Tier 1, cross-model audited (opus-5).

Entity hubs

Built around this cluster by prior promotions; listed, not built by this pass.

• entity-ij-good and entity-alan-turing — the wartime cryptanalytic root of Good-Turing.
• entity-anne-chao and entity-b-harris — the Chao(1984)←Harris(1959) lineage.
• entity-alon-orlitsky — the n·log n horizon and its matched lower bound.
• entity-bradley-efron — the shrinkage/authorship-attribution cousin (Efron & Thisted's Shakespeare estimate), cross-linked rather than central.
• entity-gregory-valiant — the corrected Valiant accounting behind the n·log n horizon's provenance.

Open threads (honest caveats, not hidden)

• Every member note but one is still seedling — the paleobiology note is now budding after its 2026-08-28 correction. The map records the current footing, not a frozen verdict — two legs still carry live [unverified-*] flags.
• Two load-bearing claims are unread at their primaries. Good-Turing's f₁/n is blocked behind five dead access routes to Good (1953); the dual-origin cryptanalytic root rests on a Tier-4 encyclopedia pointer with Fisher/Corbet/Williams (1943) unread. A single readable copy of Good (1953) would move two notes at once.
• The SQS = coverage-rarefaction identity was tested and does not hold — resolved 2026-08-28, no longer an open caveat. A direct read of both primaries (Chao & Jost 2012; Alroy's own SQS documentation) found SQS is a Monte Carlo resampling algorithm, distinct from Chao & Jost's algorithm and their closed-form equation; all three target the same coverage-standardized quantity, but they are not the same estimator. Kept here, marked closed, rather than deleted, so the record shows the caveat was checked and flipped, not dropped (claim-chao-jost-2012-calls-alroys-sqs-a-different-algorithmic-technique-from-their-closed-form-estimator).
• Where the (n−1)/n factor entered circulation is still unknown. The vault has established where it isn't (Chao 1984); the origin of the bolt-on is an open lead (Chao 1987 named in the note's own further-leads).