The cross-entropy loss that trains LLMs is I.J. Good's 1952 logarithmic proper scoring rule

The objective minimized in next-token language-model pretraining has its origin not in machine learning but in mid-century forecast verification. The logarithmic score, S_log(P, y) = −log p(y), was introduced by I.J. Good in 1952 as a strictly proper scoring rule — one whose expected value is uniquely optimized by reporting one's true probabilities, so it cannot be gamed by hedging. Per the arXiv survey Proper scoring rules for estimation and forecast evaluation (arXiv:2504.01781), it "is also known as the ignorance score in meteorology and is a strictly proper scoring rule (Good, 1952)," and "Minimizing the logarithmic score is equivalent to the well-known maximum likelihood principle."

That equivalence is the bridge. For a single observed token y, the logarithmic score −log p(y) is identical to the cross-entropy between the point mass on y and the model's predictive distribution; averaged over a corpus it is the negative log-likelihood whose minimization is maximum-likelihood estimation — the cross-entropy loss of autoregressive pretraining. The loss that makes a model calibrated is thus a probabilistic-forecasting instrument, kin to Glenn Brier's 1950 Brier score from weather-forecast verification, from which the same proper-scoring-rule family descends.

This anchors an otherwise unlinked pair in the vault: the calibration literature around LLM self-knowledge (claim-selfaware-canonical-self-knowledge-benchmark, see moc-machine-self-knowledge) and the mechanics of the training objective. The metric that scores a forecaster's honesty and the loss that trains the network are the same object under two names.