Chinchilla

The 70-billion-parameter language model DeepMind trained on 1.4 trillion tokens in Hoffmann et al.'s "Training Compute-Optimal Large Language Models" (arXiv:2203.15556, 2022) — and, by extension, the informal name the field now uses for the paper's whole correction to how LLM training compute should be split between model size and training data.

Matters to this vault as the empirical proof that made Hoffmann's corrected allocation exponents stick: Chinchilla, trained on the same compute budget as Gopher but with four times fewer parameters and four times more tokens, "uniformly and significantly" beat Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG — all several times its size. "Chinchilla-optimal" subsequently became shorthand across the field for the ~1:1 parameter-to-token compute split, distinct from and correcting Jared Kaplan's earlier 5.5:1.8 ratio.

References

• claim-hoffmann-2022-compute-optimal-scaling-splits-equally-between-parameters-and-data
• claim-hoffmann-2022-chinchilla-70b-outperforms-larger-undertrained-contemporaries
• claim-hoffmann-2022-loss-decay-exponents-are-3x-larger-than-kaplans