OpenAI publishes the first GPT paper on generative pre-training

Background

By 2018 most language systems were trained from scratch for a single task, using labelled examples such as sentences marked with their sentiment or pairs of sentences with their translations. Labelled data is costly to produce, and a model built for one task rarely carried over to another. OpenAI described its approach as a combination of two existing ideas, transformers and unsupervised pre-training.^[2]

What happened

On 11 June 2018 OpenAI published the paper *Improving Language Understanding by Generative Pre-Training*, together with a blog post titled *Improving language understanding with unsupervised learning*.^[1][2] The authors were Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever, all at OpenAI.^[1]

The method had two stages. The first was generative pre-training: a language model learned from unlabelled text by predicting the next word in long passages. The second was discriminative fine-tuning, in which the same model was adjusted on labelled data for each specific task.^[1]

Key details

Pre-training used the BooksCorpus dataset, which contains over 7,000 unique unpublished books from a variety of genres.^[1] Long continuous passages give a model practice with text that runs across many sentences, not just short snippets.

The model was a 12-layer decoder-only Transformer that uses masked self-attention, meaning each position can attend only to earlier words. Its hidden states had 768 dimensions and it used 12 attention heads. Training ran for 100 epochs on minibatches of 64 contiguous sequences of 512 tokens each.^[1]

During fine-tuning the team used task-aware input transformations, which convert each task's input into a form the model can read. As a result the architecture needed only minimal changes from one task to the next.^[1]

The result was described as task-agnostic: the same pre-trained model, with light adjustment, improved the state of the art on 9 of the 12 tasks studied. The absolute gains were 8.9% on commonsense reasoning (the Stories Cloze Test), 5.7% on question answering (RACE) and 1.5% on textual entailment (MultiNLI).^[1]

OpenAI released the code and model on GitHub as finetune-transformer-lm. The repository reproduces the ROCStories Cloze result, reported as 86.5% in the paper, and is now archived.^[3]

Impact

The paper showed that one general language model, trained on unlabelled text and then lightly adapted, could do well on many tasks that previously needed separate systems. That pattern of pre-training followed by adaptation became a common approach in later work. Because the code and model were published, other researchers could check the reported results, as the repository's reproduction of the Cloze result shows. The gains were measured on specific benchmarks, so they describe performance on those tests rather than general language ability.

The paper is an early example of that pattern. Its results were reported on a fixed set of benchmark tasks, which is worth remembering when comparing later claims about general language ability.