OpenAI publishes the GPT-3 paper

Background

A common way to adapt a pre-trained language model to a new task was to train it further on labelled examples for that task. The GPT-3 paper tested a different route: make the model much larger, then describe the task in plain text, with a few examples written into the input.^[1]

The idea that a model could learn a task from examples placed in its input, without updating its weights, was a departure from that standard practice.^[1]

What happened

On 28 May 2020 the paper *Language Models are Few-Shot Learners* was submitted to arXiv.^[1] It lists 31 OpenAI authors, led by Tom B. Brown. The list includes Jared Kaplan, Alec Radford and Ilya Sutskever, and ends with Dario Amodei.^[1]

The paper introduced GPT-3, an autoregressive language model with 175 billion parameters. The authors said this was ten times more than any previous non-sparse language model.^[1] An autoregressive model generates text one token at a time, with each token based on those before it.

Key details

The central finding was that scaling up language models greatly improves task-agnostic few-shot performance. Tasks and examples are given to the model as text, rather than taught through additional training.^[1] Few-shot means the model sees only a handful of examples in its input, and its parameters are not changed.

The paper also reported that GPT-3 could generate samples of news articles that human evaluators had difficulty distinguishing from articles written by humans.^[1]

On 7 December 2020 NeurIPS named the paper one of three NeurIPS 2020 Best Paper Award winners.^[4]

Few-shot prompting is the practical side of this finding. A user writes a short description of a task, optionally followed by a few input-and-output examples, and the model continues the text with an answer. The model's parameters are not changed in the process, so the same model can switch tasks simply by changing the prompt.^[1]

Because the paper's central argument is about scale, the parameter count is a key part of its claims.

Impact

The paper's findings were followed by a change in how the model could be used. On 11 June 2020 OpenAI announced an API for accessing its new AI models. On 22 September 2020 it agreed to license GPT-3 to Microsoft for Microsoft's own products and services.^[2][3]

The broader consequence was the idea that one large model can handle many tasks when they are described in text and illustrated with examples. That approach became a central direction for later large language models. It also depends on very large models, which are expensive to train and run.

The paper also raised a question that later work kept returning to. Text that is hard to tell apart from human writing has many legitimate uses, and it also makes it harder to know who wrote a given passage. The paper's news samples made that point concrete.^[1]