Improving Language Understanding by Generative Pre-Training (GPT)

The problem

Training a system for a language task has traditionally meant collecting labelled examples for that task, and labelled data is expensive to produce. Raw text, by contrast, is abundant. The paper asked whether a model could first learn general language from large amounts of unlabelled text, and then be adapted to a specific task with a much smaller labelled dataset. ^[1]

Pre-training is the expensive part. A single large run produces a set of learned parameters, and that result can then be adapted many times at much lower cost. This split, one costly general stage followed by many cheaper specific stages, is the idea the paper put into practice. ^[1]

The approach

OpenAI published the paper and an accompanying blog post on 11 June 2018. The authors were Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. ^[2] The model, later commonly called GPT-1, is a 12-layer Transformer decoder. A decoder reads text from left to right, so each position only sees the words before it, which makes predicting the next token a natural training task.

The model was trained with generative pre-training, meaning it learned to predict the next token (a token is a piece of text, often a word or part of one). The training corpus was BooksCorpus, a collection of over 7,000 unpublished books, chosen because it contains long stretches of continuous text. ^[1]

After pre-training, the same model was fine-tuned on each task, with only small changes to the input format. ^[1]

Fine-tuning deserves a closer look. Once the model has absorbed general patterns of language from its pre-training corpus, a new task is handled by presenting its input in a format close to the text the model already knows, then continuing training on a small labelled dataset. The model does not need a new architecture for each task, which is a large part of what made the approach attractive. ^[1]

The results

The paper evaluated the approach on a range of benchmarks. It improved the state of the art on 9 of the 12 tasks studied, including question answering, textual entailment (judging whether one sentence follows from another) and commonsense reasoning. ^[1]

Next-token prediction is a simple objective, yet it asks the model to absorb a great deal. To guess the next word well across a long passage, a model has to track grammar, the characters and topic of a story, and how an argument develops. A single training objective can therefore teach a model many skills that nobody labelled one by one. ^[1]

Why it matters

The paper set out a recipe that the GPT series followed: a decoder-only Transformer, trained to predict the next token on large unlabelled text, then adapted to tasks. Pre-training a general model and then adapting it became a central idea in later language models. ^[3]