Gopher

What Gopher is

Gopher is a large language model from DeepMind, described as a 280-billion-parameter Transformer and announced on 8 December 2021.^[1] A Transformer is the neural-network architecture that most large language models of this period are built on; a parameter is one of the numbers the network adjusts while learning. The parameter count sets how much compute and memory training and running the model require, which is why it is often used as shorthand for a model's size.

One model in a ladder of models

DeepMind did not train Gopher on its own. It trained a series of models running from 44 million to 280 billion parameters, specifically to study the effects of scale.^[1] The technical paper describes the study as covering models from tens of millions of parameters up to the 280-billion-parameter Gopher.^[2]

The ladder is the point. When many models share the same design, data pipeline and evaluation, differences in their scores can be attributed to size rather than to changes in architecture or training data. A single large model shows what is possible at that size; a family of models shows what size itself is doing.

Where scale helped

Scale improved performance most in reading comprehension, fact-checking and identifying toxic language.^[1] Each of these tasks rewards a model for having absorbed a great deal of text and for being able to point at the right piece of it. DeepMind also reported advances on the MMLU benchmark, a test of knowledge across many academic subjects, toward human expert performance.^[1]

Where it did not

Scale did not significantly improve logical reasoning and common-sense tasks.^[1] The paper states that logical and mathematical reasoning see less benefit from scale.^[2] Read together, the two findings say something narrower than "bigger is better": adding parameters made the model better at tasks that lean on stored knowledge and surface patterns in text, but did not reliably make it better at working through a problem in steps or at reasoning about everyday situations.

Two companion papers

Gopher was announced alongside two other pieces of work. One was a study classifying language-model risks into six thematic areas covering 21 risks.^[1] Rather than treating harm as a single vague worry, the study breaks it up into named categories, so that particular risks can be measured and discussed separately.

The other introduced RETRO, a Retrieval-Enhanced Transformer that matched a regular Transformer with about an order of magnitude fewer parameters.^[1] A retrieval model keeps a large store of text outside its weights and looks up relevant passages while processing an input, so some of the model's capacity comes from the store rather than from parameter count alone.

The paper and its place

The technical paper was submitted to arXiv on 8 December 2021 and last revised on 21 January 2022.^[2]

Taken as a group, the three releases answer three different questions. Gopher itself asks what happens when a language model is made very large. The risk study asks what can go wrong with such models and how those failures should be sorted. RETRO asks whether a model can reach comparable results without paying the full cost in parameters. All three questions stayed central to language-model research afterwards, and the finding that scale helps recall and reading more than reasoning became one of the standard caveats attached to large models.