Emergent Abilities of Large Language Models

The question the paper asks

Work on scaling language models had generally found that making models bigger produces steady, predictable gains: performance and sample efficiency on a wide range of downstream tasks improve as scale increases^[1]. That picture suggests a fairly simple planning rule — measure a small model, extrapolate the curve, and estimate what a much larger one will do.

The paper, submitted to arXiv on 15 June 2022 as 2206.07682, turns to a phenomenon that does not fit this picture, which the authors call emergent abilities of large language models^[1].

What counts as emergent

The authors give a deliberately narrow definition. An ability is emergent if it is not present in smaller models but is present in larger models^[1]. The defining feature is a change of kind rather than a change of degree: at some point along the size axis, something that was absent shows up.

This is a threshold-style definition, and it is worth being clear about what it does and does not claim. It is a statement about where an ability is observed, not about a special mechanism inside large models. An ability that appears abruptly with scale is still produced by the same kind of machinery as everything else the model does; what changes is that the behaviour becomes measurable.

Why the curves are hard to use

The argument the paper makes from its definition is about prediction. If an ability is absent in smaller models and present in larger ones, then its appearance cannot be foreseen simply by extrapolating the performance of the smaller models^[1]. A trend line fitted to models that all lack the ability has nothing to extend: the quantity being tracked is at zero, and a curve through zeros does not tell you when it will stop being zero.

This is what the authors mean by calling the phenomenon unpredictable. It is not a claim that scale is irrelevant — scale is precisely the variable that seems to matter — but that the usual habit of reading a capability off a smooth graph breaks down for abilities of this sort^[1].

What follows for scaling

The existence of such emergence is taken by the authors to imply that additional scaling could further expand the range of capabilities of language models^[1]. If new abilities have repeatedly come into view as models grew, there is no obvious reason to assume the process has stopped. This is a claim about possibility rather than a guarantee: it says the space of capabilities is open in a way that a pure extrapolation view would not suggest.

Publication and authorship

The paper appeared in Transactions on Machine Learning Research in 2022, and a second arXiv version is dated 26 October 2022^[1]. It has sixteen authors: Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean and William Fedus^[1].

Why it matters

The paper's contribution is largely conceptual: it names a pattern that researchers were already running into and gives it a definition precise enough to argue about^[1]. Naming it changes how scaling results get read. A flat line on a benchmark at small sizes stops being evidence that a capability is out of reach, and a jump between two model sizes becomes a data point about emergence rather than an anomaly. The definition also sets a clear target for later work, because any claimed emergent ability can be checked against it: absent before, present after^[1].