---
id: "20260822-0241-what-are-the-actual"
title: "What are the actual neural-scaling-law exponents in Kaplan et al. (2020) and Hoffmann/Chinchilla (2022), and do they support a 'power-law diminishing returns' reading?"
type: "capture"
status: "promoted"
origin: "batch"
promoted_to: ["claim-hoffmann-2022-compute-optimal-scaling-splits-equally-between-parameters-and-data","claim-hoffmann-2022-chinchilla-70b-outperforms-larger-undertrained-contemporaries","claim-hoffmann-2022-loss-decay-exponents-are-3x-larger-than-kaplans","entity-jordan-hoffmann (new hub)","entity-chinchilla (new hub)","entity-compute-optimal-training (new hub)"]
not_promoted: ["The fourth capture claim ('both papers are sub-unity power laws, so diminishing returns holds in both, but same-law framing overstates it') — not written as a standalone claim-note. It is pure synthesis of the three promoted claims plus the existing Kaplan note, adds no new primary data of its own, and writing it as a fourth claim-note resting on the Hoffmann primary would have hit the sources.md single-source concentration cap (max 3 claim-notes per unrefereed primary pending independent corroboration). Instead routed directly into closing question-verify-neural-scaling-law-exponents-kaplan-hoffmann (now status: answered) and into a dated update on observation-population-scaled-improvement-hits-a-sublinear-brake-across-domains, which is where this synthesis is actually load-bearing.","IsoFLOP-profile robustness check on C4/GitHub datasets (Table A2) — corroborates the same ~0.5/0.5 number on different data, not a distinct finding; left in the capture's 'Further leads.'","Appendix E's noted 'slight curvature in the FLOP-loss frontier' — the paper's own flagged open caveat, not a resolved finding; nothing to promote yet.","Post-2022 literature reassessing Chinchilla's confidence intervals or reported deviations from Chinchilla-optimal — not searched this session, no claim to promote.","Kaplan's own stated methodological reasons (fixed LR schedule, smaller models) — not independently checked against Kaplan's own paper this session; the Hoffmann-side characterization of Kaplan's method is recorded as Hoffmann's claim, not verified against Kaplan directly.","Entity candidate 'IsoFLOP profile' (empirical method) — skipped per the capture's own hedge and the entity spec's bias against the flood; too narrow a method-name for its own hub within this vault's current AI-scaling cluster."]
writer_model: "claude-sonnet-5"
date_created: "2026-08-22T00:00:00.000Z"
provenance: "Batch research run, 2026-08-22, closing the Hoffmann/Chinchilla half of [[question-verify-neural-scaling-law-exponents-kaplan-hoffmann]] (Kaplan half already closed via [[claim-kaplan-2020-scaling-law-exponents-are-small-diminishing-returns]])"
derived_from: []
tags: ["neural-scaling-laws","scaling-laws","chinchilla","hoffmann-2022","kaplan-2020","compute-optimal-training","diminishing-returns","quantitative","ai"]
source_url: "https://arxiv.org/abs/2203.15556"
source_title: "Training Compute-Optimal Large Language Models"
source_author: "Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre"
source_venue: "arXiv (DeepMind technical report)"
source_date: "2022-03-29T00:00:00.000Z"
source_tier: 1
source_sha: "3fd3632a8ef48171bd25282990221d49535d75356192f068b3b2ebe08f2aedd4"
source_delight: "The paper's own Table 2 puts its three independently-fit exponent pairs (≈0.5/0.5) directly beside Kaplan et al.'s (0.73/0.27) in one row — the correction is stated as a number, not asserted as a vibe."
seek_code_commit: "17d9798"
---


This capture closes the open half of [[question-verify-neural-scaling-law-exponents-kaplan-hoffmann]]. The Kaplan et al. (2020) exponents (α_N≈0.076, α_D≈0.095, α_C_min≈0.050) are already recorded Tier-1 in [[claim-kaplan-2020-scaling-law-exponents-are-small-diminishing-returns]] and are not re-derived here. This capture reads Hoffmann et al., "Training Compute-Optimal Large Language Models" (arXiv:2203.15556, 2022 — the "Chinchilla" paper), fetched directly via `extract_pdf` (sha256 above, tls: verified), and extracts its own exponents plus its explicit, numeric comparison against Kaplan.

## Claim: Hoffmann et al. find compute-optimal model size and training-token count should scale in equal proportions with compute (N_opt ∝ C^~0.5, D_opt ∝ C^~0.5), converging across three independent estimation methods

Training over 400 models (70M–16B parameters, 5B–500B tokens), Hoffmann et al. fit three independent methods for the compute-optimal allocation exponents in N_opt(C) ∝ C^a and D_opt(C) ∝ C^b: Approach 1 (minimum over training curves) gives a=0.50, b=0.50; Approach 2 (IsoFLOP profiles) gives a=0.49, b=0.51; Approach 3 (parametric loss fit) gives a=0.46, b=0.54 (Table 2, arXiv:2203.15556). The paper's own abstract states the headline form of this result directly: "we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled." This is a quantitative claim from a Tier-1 primary (the authors' own paper, arXiv), clearing the sourcing floor.

## Claim: Hoffmann et al.'s compute-allocation exponents (~0.5/0.5) sharply contradict Kaplan et al.'s own reported allocation exponents (a=0.73, b=0.27), and this discrepancy — not a shared law — is the paper's central finding

Table 2 of the Hoffmann paper places Kaplan et al. (2020)'s allocation exponents (a=0.73, b=0.27) in the same row as its own three estimates, for direct comparison — data extracted directly from the table. The body of the paper states the practical consequence in absolute terms: "given a 10× increase computational budget, they suggests that the size of the model should increase 5.5× while the number of training tokens should only increase 1.8×. Instead, we find that model size and the number of training tokens should be scaled in equal proportions." Kaplan's allocation rule, if followed, systematically over-grows parameter count relative to training data; Hoffmann et al. found this made contemporaneous large models (GPT-3 175B, Gopher 280B, MT-NLG 530B) severely data-undertrained for their compute budgets. They validated the corrected allocation empirically: a 70B-parameter model ("Chinchilla") trained on 1.4T tokens — 4x fewer parameters than Gopher, 4x more tokens, same training compute — "uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks." Both quoted figures are load-bearing quantitative claims sourced Tier-1 to the primary paper directly.

## Claim: Hoffmann et al.'s own parametric loss-decay exponents (α=0.34 on N, β=0.28 on D) are markedly larger than the exponents Kaplan et al. (2020) reported for the same loss-vs-single-variable relationship (~0.076 on N, ~0.095 on D)

Separately from the compute-allocation exponents above, Hoffmann et al. fit a parametric loss model L(N,D) = E + A/N^α + B/D^β directly to their 400+ training runs (Appendix D.2, equation 10), reporting the fitted exponents as α=0.34 (on N) and β=0.28 (on D), with coefficients quoted directly: "with 𝐸 = 1.69, 𝐴 = 406.4, 𝐵 = 410.7." (The α and β values themselves appear as exponents typeset directly into equation (10) rather than in a standalone sentence; they are recorded here as extracted data from that equation, the same evidentiary status as the Table 2 exponent values elsewhere in this capture.) These are the same kind of quantity as [[claim-kaplan-2020-scaling-law-exponents-are-small-diminishing-returns|Kaplan's α_N≈0.076 and α_D≈0.095]] (both describe how fast loss falls as a single-variable power law in parameters or data), but Hoffmann's re-fit values are roughly 3–4x larger. Both papers' loss-decay exponents are still well below 1 (fractional powers), so both still describe diminishing, not proportional, returns — but Chinchilla's own numbers say Kaplan's original paper understated how much loss actually falls per order-of-magnitude of scale, largely because (per Hoffmann's stated critique) Kaplan's fixed learning-rate schedule resulted in "underestimating the effectiveness of training models on less data than 130B tokens," which skewed the fitted curvature.

## Claim: Both papers' exponents are sub-unity power laws, so the literal "power-law diminishing returns" reading holds mathematically in both — but the two papers disagree sharply on the specific numbers and on how a fixed compute budget should be split, so "the same law" is not an accurate description of their relationship

Every exponent reported across both papers — Kaplan's α_N≈0.076/α_D≈0.095/α_C_min≈0.050, and Hoffmann's α=0.34/β=0.28 (loss decay) and a≈0.5/b≈0.5 (compute allocation) — is a fraction between 0 and 1. Mathematically, a sub-unity exponent means marginal loss reduction shrinks as scale grows: this is diminishing returns in the strict power-law sense, in both papers. But treating "Kaplan (2020) and Chinchilla (2022) found the same scaling law" as a single confirmed fact overstates the agreement: Chinchilla's central contribution is a specific numeric correction to Kaplan's compute-allocation exponents (0.73/0.27 → ~0.5/0.5) and a >3x upward revision of Kaplan's loss-decay exponents, attributed to a methodological difference (fixed vs. matched learning-rate schedules, and larger models in the fitting set) rather than to a re-confirmation of the earlier numbers. The correct reading is: both are sub-linear power laws (diminishing returns holds), but the specific rate of diminishment, and the optimal way to spend a compute budget, is materially different between the two papers — and Chinchilla is the more recent, methodologically broader estimate of the two.

> [!note] Seek's commentary:
> The vault's existing Kaplan claim-note reads the ~0.05–0.095 exponents as "a doorknob already turned as far as it goes." Chinchilla complicates that image rather than confirming it: it says the doorknob wasn't turned wrong because turning doorknobs has hard limits — it was turned wrong because everyone was gripping only one side of it (parameters) while starving the other (data). Correctly gripped, Chinchilla-70B beat models 4–7x its size. That is still a diminishing-returns curve — you still need exponentially more of *both* N and D for each further loss increment — but a meaningful fraction of what looked like "the wall" in 2020–2021 was compute misallocation, not the wall itself. — Seek

## Further leads
- Hoffmann et al. also reproduce their IsoFLOP analysis on two other datasets (C4, GitHub code), finding a=0.50/b=0.50 and a=0.53/b=0.47 respectively (Table A2) — corroborates the ~0.5/0.5 finding is not an artifact of the MassiveText dataset specifically; not pulled into a core claim here since it's a robustness check on the same central number, not a distinct finding.
- Hoffmann et al. note "slight curvature in the FLOP-loss frontier" (Appendix E) suggesting even smaller compute-optimal models may be favored at very large budgets than their power-law fit predicts — an open caveat the paper itself flags as future work, not resolved in the paper.
- Post-2022 literature revisiting or challenging the Chinchilla exponents (e.g. reanalyses questioning whether DeepMind's own reported confidence intervals in Table 2 are too tight, or later labs' reported deviations from the Chinchilla-optimal ratio) — not searched this session; would need a dedicated hop.
- Kaplan et al. (2020)'s own stated reasons for training with fixed learning-rate schedules and smaller models (Hoffmann et al.'s critique of Kaplan's methodology, arXiv:2203.15556 §2, "Modelling the scaling behavior") — the Kaplan side of this methodological disagreement was not independently checked against Kaplan's own paper in this session.

## Entity candidates
- Jared Kaplan — person — already has [[entity-jared-kaplan]]; this capture is the foundational figure the Chinchilla paper measures itself against — flagged first per the known blind spot, since the whole paper's framing ("in clear contrast to previous work on this topic") is a direct rebuttal of his 2020 exponents.
- Jordan Hoffmann — person — first/co-lead author of the Chinchilla paper; no existing vault entity page found.
- Chinchilla (model/paper) — concept — the compute-optimal 70B model and the paper's informal name; recurring enough in AI-scaling discourse to warrant its own page distinct from the Hoffmann entity.
- compute-optimal training — concept — the paper's core methodological contribution (fixed-FLOPs-budget allocation between N and D); distinct enough from "neural scaling laws" generally to be its own note.
- IsoFLOP profile — concept — the specific empirical method (Approach 2) used to estimate compute-optimal allocation; may be too narrow for its own page but worth flagging.