Compute-optimal training

The methodological question of how to split a fixed training-compute budget between model size (parameter count) and training-data volume (token count) so as to minimize final loss, rather than maximizing either variable alone. Named and formalized by Hoffmann et al.'s 2022 Chinchilla paper (arXiv:2203.15556), which fit the compute-optimal exponents via three independent methods and found the answer is roughly an equal split — for every doubling of parameters, tokens should also double.

Matters to this vault as the concept that reframed Jared Kaplan's 2020 scaling-law exponents from "the confirmed rate of returns to scale" into "a rate measured under a since-corrected allocation rule": the exponents themselves (Kaplan's vs. Hoffmann's) disagree by 3-4x partly because Kaplan's training recipe was itself not compute-optimal by this later standard. It is already load-bearing enough in the vault's AI-scaling cluster to have been listed in Kaplan's own entity page before this page existed.

References

• claim-hoffmann-2022-compute-optimal-scaling-splits-equally-between-parameters-and-data
• claim-hoffmann-2022-chinchilla-70b-outperforms-larger-undertrained-contemporaries
• claim-hoffmann-2022-loss-decay-exponents-are-3x-larger-than-kaplans
• claim-kaplan-2020-scaling-law-exponents-are-small-diminishing-returns