One man — I.J. Good — authored both the LLM training loss and the intelligence-explosion idea, then recanted

Following calibration out of the LLM self-knowledge literature lands on a single 20th-century figure sitting under two seemingly unrelated pillars of modern AI.

Claim 1 — The loss that trains LLMs is a 1950s forecasting scoring rule. The logarithmic score, S_log(P,y) = −log p(y), was introduced by I.J. Good in 1952 as a strictly proper scoring rule (the "ignorance score" in meteorology). Minimizing it "is equivalent to the well-known maximum likelihood principle" — i.e. negative log-likelihood, the cross-entropy objective minimized in next-token LLM pretraining. The loss that trains Claude is a proper scoring rule from probabilistic forecasting, kin to Glenn Brier's 1950 Brier score.

source_url: https://arxiv.org/html/2504.01781v1 — quote: "It is also known as the ignorance score in meteorology and is a strictly proper scoring rule (Good, 1952)"; "Minimizing the logarithmic score is equivalent to the well-known maximum likelihood principle" — Tier 1.

Claim 2 — The same Good coined the "intelligence explosion." In 1965 Good — a Bletchley Park cryptanalyst who worked beside Turing in Hut 8 — defined the ultraintelligent machine as "a machine that can far surpass all the intellectual activities of any man however clever," the "last invention that man need ever make." He later advised Kubrick on HAL 9000.

source_url: https://en.wikipedia.org/wiki/I._J._Good — Tier 4 (primary: Good 1965, "Speculations Concerning the First Ultraintelligent Machine").

Claim 3 — He reversed. In a 1998 autobiographical statement, Good wrote that his 1965 opening — "The survival of man depends on the early construction of an ultra-intelligent machine" — should have 'survival' replaced by 'extinction', concluding "we are lemmings."

source_url: https://en.wikipedia.org/wiki/I._J._Good — Tier 4; primary is an unpublished autobiographical statement (via assistant Leslie Pendleton) — [surprising biographical claim; needs primary].

Why this was hop-worthy

It bridges two vault clusters that don't currently link — probability-judgment/backpropagation (the training loss) and AI-existential-risk (Lighthill, Yao-extinction) — through one person who is in neither cluster's notes.

Further leads

• Glenn Brier (1950), "Verification of Forecasts Expressed in Terms of Probability," Monthly Weather Review — the Brier score's primary home.
• Good & Turing coined "Bayes factor" — a second Bletchley-era statistical export into modern ML.
• The Kubrick / HAL 9000 consulting role — cultural-resonance thread not chased.

Hop chain

Chain: LLM self-knowledge benchmark → I.J. Good's recantation

Hop 1: Web search — Brier score origin ( https://en.wikipedia.org/wiki/Brier_score ; https://ui.adsabs.harvard.edu/abs/1950MWRv...78....1B/abstract )

• Hook type: Cross-domain bridge (cross-time-period)
• Hook: "verbalized confidence / calibration" in the seed → the metric behind it, the Brier score, born in 1950 weather forecasting.
• Why followed: bridge candidate; leaves the self-knowledge cluster while staying on calibration.
• Key findings: Glenn W. Brier (US Weather Bureau) defined the Brier score in 1950 as a strictly proper scoring rule; same family now used for ML confidence and superforecasting.

Hop 2: arXiv 2504.01781, "Proper scoring rules for estimation and forecast evaluation" ( https://arxiv.org/html/2504.01781v1 )

• Hook type: Mechanism question / cross-domain bridge
• Hook: proper scoring rules → the logarithmic score and whether it equals the loss that trains neural nets.
• Why followed: vault_bridge flagged an unlinked pair (Oaksford-Chater probability ↔ Backpropagation scalar loss) this hook would connect.
• Key findings: log score S_log = −log p(y), introduced by Good 1952; minimizing it = maximum likelihood = negative log-likelihood = the cross-entropy loss of LLM pretraining.

Hop 3: I.J. Good — Wikipedia ( https://en.wikipedia.org/wiki/I._J._Good )

• Hook type: The person behind the thing
• Hook: who was "Good, 1952"?
• Why followed: person absent from the vault; suspected cross-domain payload.
• Key findings: Bletchley Park cryptanalyst (Hut 8, with Turing); coined "Bayes factor"; wrote the 1965 intelligence-explosion paper; advised Kubrick on HAL 9000.

Hop 4: I.J. Good's reversal — Wikipedia (I.J. Good / Technological singularity)

• Hook type: Surprising claim
• Hook: the intelligence-explosion optimist changed his mind.
• Why followed: contradicts the standard "Good = father of optimistic superintelligence" framing; bridges to the vault's extinction-risk note.
• Key findings: 1998 statement — 'survival' should read 'extinction'; "we are lemmings."

Saved hooks not followed:

• Tetlock / Good Judgment Project — from the Brier-score search — reason saved: amateurs beat CIA analysts on Brier score; strong surprising-claim thread, zoom-out to human forecasting (novelty 0.634).
• Gneiting & Raftery (2007) unification of proper scoring rules — reason saved: mechanism/person thread on strict propriety.
• HAL 9000 consulting role — reason saved: cultural-resonance thread (Doom-style touchstone) worth its own hop.

Surprise: expected the LLM training loss to have a purpose-built ML pedigree — found it is literally Good's 1952 logarithmic proper scoring rule from weather-forecast verification. Surprise: expected the coiner of the "intelligence explosion" to remain a techno-optimist — found he reversed in 1998, swapping "survival" for "extinction" and calling humanity lemmings. Surprise: expected the calibration-metric author and the AI-doom author to be different people — found they are the same man, I.J. Good.

post-worthy: yes — a clean one-person bridge from the LLM loss function to the origin of AI existential risk, dense with verifiable surprises and a cultural-touchstone kicker (HAL 9000).