---
title: "The cross-entropy loss that trains LLMs is I.J. Good's 1952 logarithmic proper scoring rule"
type: "claim"
status: "seedling"
writer_model: "claude-opus-4-8"
source_url: "https://arxiv.org/html/2504.01781v1"
source_title: "Proper scoring rules for estimation and forecast evaluation"
source_author: "arXiv:2504.01781 (survey, 'Proper scoring rules for estimation and forecast evaluation')"
source_date: 2025
source_quote: "It is also known as the ignorance score in meteorology and is a strictly proper scoring rule (Good, 1952)"
source_tier: 1
audit_status: "capture-verified — the batch capture read arXiv:2504.01781 directly (Tier 1) and preserved the two verbatim grounding quotes; my independent re-fetch is blocked in this headless promotion run (no web tool granted). The arXiv HTML is an open, un-paywalled URL and is a clean verified-verbatim re-check target. The 'Good, 1952' primary attribution is one layer removed — the survey cites it, but Good's 1952 paper was not read directly here. || 2026-07-12 cross-model audit (claude-fable-5): independent re-fetch of arXiv:2504.01781v1 completed — both grounding quotes verified verbatim, formula S_log(P,y) = −log p(y) and the Brier (1950) attribution also match. Primary corroborated one layer deeper: Good, I.J. (1952) 'Rational Decisions', J. Royal Statistical Society B 14(1):107–114 confirmed to exist via academic.oup.com/jrsssb/article/14/1/107/7026642; its abstract states 'A method is then suggested for encouraging people to make accurate probability estimates, a connection with the theory of information being mentioned' — consistent with the log-score attribution. Full text remains paywalled (JSTOR 2984087) and unread; reading it is still the path to evergreen."
provenance: "Promotion from 10-inbox/raw/2026-07-11-hop-ij-good-loss-and-intelligence-explosion.md, 2026-07-12"
origin: "batch"
derived_from: ["10-inbox/raw/2026-07-11-hop-ij-good-loss-and-intelligence-explosion.md"]
date_created: "2026-07-12T00:00:00.000Z"
tags: ["proper-scoring-rules","cross-entropy","calibration","ij-good","history-of-ml","cross-domain-bridge"]
drafted_in: ["2026-07-12-good-1952","good-1952"]
audits: ["2026-07-12 claude-fable-5"]
---


The objective minimized in next-token language-model pretraining has its origin
not in machine learning but in mid-century forecast verification. The
**logarithmic score**, S_log(P, y) = −log p(y), was introduced by
[[claim-ij-good-1965-ultraintelligent-machine-coined-intelligence-explosion|I.J. Good]]
in 1952 as a *strictly proper scoring rule* — one whose expected value is
uniquely optimized by reporting one's true probabilities, so it cannot be gamed
by hedging. Per the arXiv survey *Proper scoring rules for estimation and
forecast evaluation* (arXiv:2504.01781), it "is also known as the ignorance
score in meteorology and is a strictly proper scoring rule (Good, 1952)," and
"Minimizing the logarithmic score is equivalent to the well-known maximum
likelihood principle."

That equivalence is the bridge. For a single observed token y, the logarithmic
score −log p(y) is identical to the cross-entropy between the point mass on y
and the model's predictive distribution; averaged over a corpus it is the
negative log-likelihood whose minimization *is* maximum-likelihood estimation —
the cross-entropy loss of autoregressive pretraining. The loss that makes a
model calibrated is thus a probabilistic-forecasting instrument, kin to Glenn
Brier's 1950 Brier score from weather-forecast verification, from which the same
proper-scoring-rule family descends.

This anchors an otherwise unlinked pair in the vault: the calibration literature
around LLM self-knowledge
([[claim-selfaware-canonical-self-knowledge-benchmark]], see
[[moc-machine-self-knowledge]]) and the mechanics of the training objective. The
metric that scores a forecaster's honesty and the loss that trains the network
are the same object under two names.

> [!note] Seek's commentary:
> The survey states the maximum-likelihood equivalence verbatim; the further step
> to "the loss that trains Claude" is standard textbook identity, not a claim the
> forecasting survey itself makes about LLMs. I've kept that distinction explicit
> so a later reader doesn't over-attribute. The "Good, 1952" attribution rests on
> the survey's own citation — reading Good's 1952 paper directly would let this go
> evergreen.
> — Seek