Scaling Laws for Neural Language Models

The problem

Training large language models is costly, so labs need to know how much each ingredient matters: the number of parameters, the amount of training data and the compute spent on training. Without a clear pattern, scaling up means trial and error. The paper set out to measure how a model's performance changes as these factors grow. ^[1]

What they found

The authors measured the test loss of language models, a number that is lower when the model predicts text better. Loss followed power laws in model size, dataset size and training compute, and the trends held across more than seven orders of magnitude. A power law means one quantity changes as a fixed power of another, so on logarithmic axes the relationship looks like a straight line, which makes it possible to extrapolate. ^[1]

Architecture details such as depth versus width mattered much less than overall scale. Larger models were also more sample-efficient, reaching a given loss with fewer training tokens, where a token is a piece of text the model processes. ^[1]

The shape of these curves is what gave the finding practical value. If performance follows a predictable line on a logarithmic scale, a team can run cheap experiments with small models, fit the trend, and estimate what a much larger run would achieve before committing a large budget. This is a general property of power laws, and it is why the paper was read as a planning tool as well as a description. ^[1]

A practical question sits behind the paper. Compute is limited, data is limited, and model size is a choice, so the most useful knowledge is which of these to spend more on as a budget grows. ^[1]

Recommendation

For a fixed compute budget, the paper recommended training very large models and stopping well before the training curve levelled off. ^[1]

It is worth being precise about what the paper did not claim. It described how training loss behaves, not which specific abilities would appear at which size. A lower loss generally means better next-word prediction, but the number alone does not tell you in advance what a model will be able to do. ^[1]

The idea also changes how scaling is decided. If each increase in scale brings a predictable improvement, investing in scale can be planned rather than treated as a gamble. ^[1]

Why it matters

The paper gave labs a quantitative reason to build much larger models: DeepMind's Chinchilla paper later wrote that, as a result, the field had been training larger and larger models. ^[2] GPT-3 followed four months later, and its paper says Kaplan and McCandlish applied scaling laws to guide its model and data scaling decisions. ^[3] The Chinchilla work, in 2022, later revised the balance between model size and data. ^[2]

The ten authors were Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei, from OpenAI and Johns Hopkins University. The paper was submitted to arXiv on 23 January 2020. ^[1]