Scaling laws

What it is

A scaling law is an empirical pattern, a rule found by measuring rather than one derived from theory. For language models, the pattern is that test loss, a measure of how badly the model predicts text it has not seen, falls smoothly as a power law when three things grow: the number of parameters, the size of the training dataset measured in tokens, and the amount of training compute.^[1]

How to read it

If you plot loss against model size on logarithmic axes, a power law usually appears as a roughly straight line. This means a trend measured on small models can be extended to estimate how larger models will behave. This is the idea behind predicting the result of a big training run from small experiments.^[1]

Because the relationship is smooth, a small study can reveal the trend well before a costly run is finished.

Where it came from

Kaplan and colleagues at OpenAI measured these trends across more than seven orders of magnitude, in a paper submitted in January 2020. They concluded that, for a fixed compute budget, it pays to train very large models.^[1]

In March 2022, DeepMind's Chinchilla paper by Hoffmann and colleagues revised that recipe. It argued that model size and the number of training tokens should grow in roughly equal proportion.^[2] To test this, the team trained a 70-billion-parameter model, Chinchilla, on 1.4 trillion tokens. With the same compute as the 280-billion-parameter Gopher, it outperformed Gopher.^[2]

The three quantities are linked. A larger model needs more data to be trained well, and training either one takes more compute. Scaling laws describe how loss responds when these grow together, which is why they treat parameters, data and compute as one planning problem.

Why the word law

The word law is used loosely here. Scaling laws are empirical regularities that held across the range the researchers measured. They are useful for planning, but they are not guarantees, and they are not physical laws derived from first principles.

Why it matters

Scaling laws made model building more like engineering. Labs could predict the result of a large training run from small experiments and plan their compute budgets in advance.^[1]

These laws also help explain why AI developers have spent so heavily on ever-larger clusters of chips. Investments in compute have reliably delivered better capabilities, and scaling laws put numbers on those returns.^[3]

Limits

Scaling laws describe training loss, the quantity the model optimizes during training. They are not reliable predictors of performance on specific tasks, which can improve suddenly and unexpectedly.^[3] A fitted law is also a summary of the range that was measured. Extending it far beyond that range makes the conclusions less certain, and the balance between size and data may shift as methods change.