Training Compute-Optimal Large Language Models (Chinchilla)

The question

Training a language model costs compute, and a fixed compute budget can be spent in two places: on a larger model, with more parameters, or on more training data, counted in tokens. The prevailing practice leaned heavily toward the first — models grew quickly while the amount of text they were trained on stayed comparatively modest. The paper asks how a given budget should actually be divided between the two.

The experiment

Rather than train one model and argue from it, Jordan Hoffmann and colleagues at DeepMind trained more than 400 language models, ranging from 70 million to over 16 billion parameters and trained on 5 to 500 billion tokens ^[1]. The paper was submitted to arXiv on 29 March 2022 and published at NeurIPS 2022 ^[1]. Because both size and data varied across hundreds of runs, the authors could compare configurations that consumed the same amount of compute and ask which of them reached the lowest loss — training that is "compute-optimal" in their sense. Two terms matter here: a parameter is a learned weight inside the network, and a token is a piece of text, roughly a word or part of a word, that the model is trained to predict.

The finding

The answer came out as a scaling rule: for compute-optimal training, model size and the number of training tokens should be increased in equal proportion ^[1]. Double the budget, and both the model and its dataset should roughly double, rather than the model alone. The rule carried an uncomfortable corollary: most large models of the time were undertrained on data ^[1]. They had many parameters but had seen comparatively little text.

The paper put numbers on the gap. Taking the compute budget used to train Gopher, a 280-billion-parameter model, it estimated that an optimal model for that budget would be 4 times smaller while trained on 4 times more tokens ^[1].

Chinchilla

To test the prediction rather than only fit it, the authors trained Chinchilla: 70 billion parameters on 1.4 trillion tokens, at the same compute cost as the 280-billion-parameter Gopher ^[1]. Chinchilla outperformed Gopher, GPT-3 and other larger models on a wide range of tasks ^[1]. The model that won was about a quarter the size of its comparison point, but had been given several times as much text to learn from.

Why it mattered

The result pointed labs toward smaller models trained on far more data, which are also cheaper to run ^[2]. Its influence showed up in later work: Meta's LLaMA paper, published in 2023, cites the finding that for a given compute budget the best performance comes from smaller models trained on more data, and states a preference for models that are fastest at inference ^[2].

Scope

The sweep covered models from 70 million to over 16 billion parameters, and the prediction was tested at 70 billion ^[1]. The rule concerns a single decision — how to split training compute between model size and data — but that decision determines what a given budget actually buys.