Hoffmann et al. (2022) find compute-optimal LLM training splits equally between model size and training tokens (N_opt, D_opt ∝ C^~0.5), contradicting Kaplan et al.'s (2020) 0.73/0.27 split

neural-scaling-lawschinchillahoffmann-2022compute-optimal-trainingkaplan-2020diminishing-returnsquantitativeai

Hoffmann et al., "Training Compute-Optimal Large Language Models" (arXiv:2203.15556, 2022 — the "Chinchilla" paper), trained over 400 models (70M–16B parameters, 5B–500B tokens) to fit the compute-optimal allocation exponents in N_opt(C) ∝ C^a and D_opt(C) ∝ C^b. Three independent estimation methods converge closely: minimum-over-training-curves gives a=0.50, b=0.50; IsoFLOP profiles give a=0.49, b=0.51; a parametric loss fit gives a=0.46, b=0.54 (Table 2). The paper's abstract states the headline directly: "we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled."

This directly contradicts Kaplan et al.'s (2020) own reported allocation exponents (a=0.73, b=0.27), which the paper places in the same Table 2 row for comparison. Hoffmann et al. spell out the practical gap: given a 10x increase in compute budget, Kaplan's rule implies "the size of the model should increase 5.5x while the number of training tokens should only increase 1.8x," whereas the corrected rule scales both equally. Kaplan's allocation rule, followed at scale, systematically over-grows parameter count relative to training data — the paper's stated diagnosis for why contemporaneous large models (GPT-3 175B, Gopher 280B, MT-NLG 530B) were data-undertrained for their compute budgets. This is a distinct exponent family from Hoffmann's own loss-decay exponents — this note concerns the allocation split of a fixed compute budget, not the rate at which loss falls.