Language Models are Unsupervised Multitask Learners (GPT-2)

The problem

A common approach was to build and tune a language system for one task at a time. The GPT-2 paper asked a different question: if a language model is trained on a large and varied body of text, can it perform many tasks without being specially trained for them? A prompt is the text a user supplies to steer the model, such as the start of a passage or a question, and the paper tested what a model could do from prompts alone. ^[1]

The approach

OpenAI published the paper on 14 February 2019. Its authors were Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. The model, GPT-2, is a Transformer decoder with 1.5 billion parameters, about ten times GPT-1. It is trained to predict the next word, scaled up in both model size and data. ^[1]

Its training data, WebText, is about 40 GB of text from 8 million web pages. Those pages were linked from Reddit posts that had received at least 3 karma, Reddit's score system for posts. ^[1]

This kind of use is often called zero-shot: the model receives a description or a prompt and produces an answer with no examples and no extra training for that task. The paper's notable point was that such behaviour emerged from ordinary next-word training on a large text collection, rather than from being designed in for each task. ^[1]

The results

Without any task-specific training, the model performed reading comprehension, translation, summarisation and question answering to some degree, simply when prompted. ^[1]

Reddit's karma is a score that users earn as other users upvote their posts. Requiring a minimum score was therefore a rough filter for pages that people had found worth linking to. ^[1]

Language models of this kind produce text by repeatedly choosing a likely next word, so their abilities show up through what they write when prompted. Reading the output is therefore the main way to judge what a model can do. ^[1]

Staged release

OpenAI initially withheld the full model, citing concerns about misuse, and released it in stages. The full 1.5-billion-parameter model was released on 5 November 2019. ^[2]

The release decision raised questions that apply to any powerful model: how to weigh openness against the chance of misuse, and whether releasing a model in stages gives people time to understand it. Those questions are about how such systems are shared, not only about this one model. ^[2]

The paper is a useful reference point for a pattern that later work built on: one general model trained on broad text, steered with prompts rather than retrained for each task. ^[1]

Why it matters

The paper showed that scaling up next-word prediction can produce general abilities. ^[1] It also started a public debate about releasing powerful models. ^[3]