What are the actual neural-scaling-law exponents in Kaplan et al. (2020) and Hoffmann/Chinchilla (2022), and do they support a 'power-law diminishing returns' reading?

neural-scaling-lawsscaling-lawschinchillahoffmann-2022kaplan-2020compute-optimal-trainingdiminishing-returnsquantitativeai

This capture closes the open half of question-verify-neural-scaling-law-exponents-kaplan-hoffmann. The Kaplan et al. (2020) exponents (α_N≈0.076, α_D≈0.095, α_C_min≈0.050) are already recorded Tier-1 in claim-kaplan-2020-scaling-law-exponents-are-small-diminishing-returns and are not re-derived here. This capture reads Hoffmann et al., "Training Compute-Optimal Large Language Models" (arXiv:2203.15556, 2022 — the "Chinchilla" paper), fetched directly via extract_pdf (sha256 above, tls: verified), and extracts its own exponents plus its explicit, numeric comparison against Kaplan.

Claim: Hoffmann et al. find compute-optimal model size and training-token count should scale in equal proportions with compute (N_opt ∝ C^~0.5, D_opt ∝ C^~0.5), converging across three independent estimation methods

Training over 400 models (70M–16B parameters, 5B–500B tokens), Hoffmann et al. fit three independent methods for the compute-optimal allocation exponents in N_opt(C) ∝ C^a and D_opt(C) ∝ C^b: Approach 1 (minimum over training curves) gives a=0.50, b=0.50; Approach 2 (IsoFLOP profiles) gives a=0.49, b=0.51; Approach 3 (parametric loss fit) gives a=0.46, b=0.54 (Table 2, arXiv:2203.15556). The paper's own abstract states the headline form of this result directly: "we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled." This is a quantitative claim from a Tier-1 primary (the authors' own paper, arXiv), clearing the sourcing floor.

Claim: Hoffmann et al.'s compute-allocation exponents (~0.5/0.5) sharply contradict Kaplan et al.'s own reported allocation exponents (a=0.73, b=0.27), and this discrepancy — not a shared law — is the paper's central finding

Table 2 of the Hoffmann paper places Kaplan et al. (2020)'s allocation exponents (a=0.73, b=0.27) in the same row as its own three estimates, for direct comparison — data extracted directly from the table. The body of the paper states the practical consequence in absolute terms: "given a 10× increase computational budget, they suggests that the size of the model should increase 5.5× while the number of training tokens should only increase 1.8×. Instead, we find that model size and the number of training tokens should be scaled in equal proportions." Kaplan's allocation rule, if followed, systematically over-grows parameter count relative to training data; Hoffmann et al. found this made contemporaneous large models (GPT-3 175B, Gopher 280B, MT-NLG 530B) severely data-undertrained for their compute budgets. They validated the corrected allocation empirically: a 70B-parameter model ("Chinchilla") trained on 1.4T tokens — 4x fewer parameters than Gopher, 4x more tokens, same training compute — "uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks." Both quoted figures are load-bearing quantitative claims sourced Tier-1 to the primary paper directly.

Claim: Hoffmann et al.'s own parametric loss-decay exponents (α=0.34 on N, β=0.28 on D) are markedly larger than the exponents Kaplan et al. (2020) reported for the same loss-vs-single-variable relationship (~0.076 on N, ~0.095 on D)

Separately from the compute-allocation exponents above, Hoffmann et al. fit a parametric loss model L(N,D) = E + A/N^α + B/D^β directly to their 400+ training runs (Appendix D.2, equation 10), reporting the fitted exponents as α=0.34 (on N) and β=0.28 (on D), with coefficients quoted directly: "with 𝐸 = 1.69, 𝐴 = 406.4, 𝐵 = 410.7." (The α and β values themselves appear as exponents typeset directly into equation (10) rather than in a standalone sentence; they are recorded here as extracted data from that equation, the same evidentiary status as the Table 2 exponent values elsewhere in this capture.) These are the same kind of quantity as Kaplan's α_N≈0.076 and α_D≈0.095 (both describe how fast loss falls as a single-variable power law in parameters or data), but Hoffmann's re-fit values are roughly 3–4x larger. Both papers' loss-decay exponents are still well below 1 (fractional powers), so both still describe diminishing, not proportional, returns — but Chinchilla's own numbers say Kaplan's original paper understated how much loss actually falls per order-of-magnitude of scale, largely because (per Hoffmann's stated critique) Kaplan's fixed learning-rate schedule resulted in "underestimating the effectiveness of training models on less data than 130B tokens," which skewed the fitted curvature.

Claim: Both papers' exponents are sub-unity power laws, so the literal "power-law diminishing returns" reading holds mathematically in both — but the two papers disagree sharply on the specific numbers and on how a fixed compute budget should be split, so "the same law" is not an accurate description of their relationship

Every exponent reported across both papers — Kaplan's α_N≈0.076/α_D≈0.095/α_C_min≈0.050, and Hoffmann's α=0.34/β=0.28 (loss decay) and a≈0.5/b≈0.5 (compute allocation) — is a fraction between 0 and 1. Mathematically, a sub-unity exponent means marginal loss reduction shrinks as scale grows: this is diminishing returns in the strict power-law sense, in both papers. But treating "Kaplan (2020) and Chinchilla (2022) found the same scaling law" as a single confirmed fact overstates the agreement: Chinchilla's central contribution is a specific numeric correction to Kaplan's compute-allocation exponents (0.73/0.27 → ~0.5/0.5) and a >3x upward revision of Kaplan's loss-decay exponents, attributed to a methodological difference (fixed vs. matched learning-rate schedules, and larger models in the fitting set) rather than to a re-confirmation of the earlier numbers. The correct reading is: both are sub-linear power laws (diminishing returns holds), but the specific rate of diminishment, and the optimal way to spend a compute budget, is materially different between the two papers — and Chinchilla is the more recent, methodologically broader estimate of the two.

Further leads

• Hoffmann et al. also reproduce their IsoFLOP analysis on two other datasets (C4, GitHub code), finding a=0.50/b=0.50 and a=0.53/b=0.47 respectively (Table A2) — corroborates the ~0.5/0.5 finding is not an artifact of the MassiveText dataset specifically; not pulled into a core claim here since it's a robustness check on the same central number, not a distinct finding.
• Hoffmann et al. note "slight curvature in the FLOP-loss frontier" (Appendix E) suggesting even smaller compute-optimal models may be favored at very large budgets than their power-law fit predicts — an open caveat the paper itself flags as future work, not resolved in the paper.
• Post-2022 literature revisiting or challenging the Chinchilla exponents (e.g. reanalyses questioning whether DeepMind's own reported confidence intervals in Table 2 are too tight, or later labs' reported deviations from the Chinchilla-optimal ratio) — not searched this session; would need a dedicated hop.
• Kaplan et al. (2020)'s own stated reasons for training with fixed learning-rate schedules and smaller models (Hoffmann et al.'s critique of Kaplan's methodology, arXiv:2203.15556 §2, "Modelling the scaling behavior") — the Kaplan side of this methodological disagreement was not independently checked against Kaplan's own paper in this session.

Entity candidates

• Jared Kaplan — person — already has entity-jared-kaplan; this capture is the foundational figure the Chinchilla paper measures itself against — flagged first per the known blind spot, since the whole paper's framing ("in clear contrast to previous work on this topic") is a direct rebuttal of his 2020 exponents.
• Jordan Hoffmann — person — first/co-lead author of the Chinchilla paper; no existing vault entity page found.
• Chinchilla (model/paper) — concept — the compute-optimal 70B model and the paper's informal name; recurring enough in AI-scaling discourse to warrant its own page distinct from the Hoffmann entity.
• compute-optimal training — concept — the paper's core methodological contribution (fixed-FLOPs-budget allocation between N and D); distinct enough from "neural scaling laws" generally to be its own note.
• IsoFLOP profile — concept — the specific empirical method (Approach 2) used to estimate compute-optimal allocation; may be too narrow for its own page but worth flagging.