Language Models are Few-Shot Learners (GPT-3)

The problem

Fine-tuning is the usual way to adapt a language model to a particular job. It adjusts the model's internal weights using a dataset collected for that job, so each new task traditionally meant new data and another training run. The GPT-3 paper asked a different question: how far can one model go if it is only shown a description of a task and a few examples inside its prompt, with no further training?^[1]

How it worked

The model, called GPT-3, is a Transformer decoder with 175 billion parameters, more than 100 times the size of GPT-2.^[1] Parameters are the numbers inside a neural network that training adjusts. A larger count gives the model more capacity to capture patterns in text, but it also makes training and running the model far more expensive.

It was trained on a mix of filtered Common Crawl, WebText2, two book corpora and English Wikipedia, about 300 billion training tokens in total.^[1] A token is the unit of text a model reads and writes, often a whole word or a fragment of one. Mixing web pages, books and an encyclopedia exposes the model to many different styles of writing.

The word "few-shot" describes how the model is used. The prompt contains a handful of worked examples before the new input, much as a teacher might show a student a few solved problems before asking for one more. The model is not given explicit rules. It infers the pattern from the description and the examples.

The key point is what happens at the moment a task is given. The description and examples are written into the prompt, and the model continues the text in a way that completes the task. The weights of the model do not change.

Results

The main finding was that, without fine-tuning, the model could handle many tasks from a description and a few examples in the prompt. This ability grew as the model got larger, so scale was part of the result.^[1]

The paper also tested whether the model could write news articles. Human evaluators had difficulty telling GPT-3's articles apart from articles written by people.^[1]

Why it matters

The paper showed that one large model, steered by the text in its prompt, could take on many different kinds of tasks without task-specific training.^[1][3] For people building software, this changed where the work sits. Instead of gathering a dataset and training a model for each task, much of the effort could move to writing a clear prompt that describes the job and shows a few examples.

What followed

OpenAI offered GPT-3 through a paid API, announced on 11 June 2020, rather than releasing the model's weights.^[2] Developers and companies could use a very large model as a service: they sent it prompts and received text back, without training or hosting a model themselves.^[2]