GPT-1

What it is

GPT-1 is the first model in OpenAI's GPT (generative pre-trained transformer) series. It is a language model: a program that learns to predict text, and can be adapted to other language tasks afterwards. The model was published on 11 June 2018 in the paper "Improving Language Understanding by Generative Pre-Training", with a blog post titled "Improving language understanding with unsupervised learning". It was only later that people began calling it GPT-1 ^[1][2].

The authors were Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever ^[1].

Origins

OpenAI described the approach as combining transformers with unsupervised pre-training. The method has two stages. First, the model is trained as a generative language model on unlabelled text. Second, it is discriminatively fine-tuned on each specific task ^[2].

Its architecture is a 12-layer, decoder-only Transformer with masked self-attention, 768-dimensional states and 12 attention heads ^[1]. Self-attention lets each word in a passage weigh the other words around it when building its representation. The "masked" part means that, when predicting the next word, the model cannot look at words that come after it. Decoder-only means the model is built only from this generating half of the original Transformer design.

Pre-training used minibatches of 64 contiguous sequences of 512 tokens, run for 100 epochs ^[1]. A token is a unit of text, often a word or part of a word. An epoch is one full pass through the training data.

The pre-training data was the BooksCorpus dataset, which contains over 7,000 unique unpublished books. The authors chose it because it contains long stretches of continuous text, which suits a model that learns by predicting the next word ^[1].

Main features

The most distinctive feature was how little each task changed the architecture. Fine-tuning used task-aware input transformations, so the model needed only minimal changes for each task ^[1]. Instead of building a new network for every problem, a user could feed a task into the same pre-trained model in a structured form.

Results were broad. GPT-1 improved the state of the art on 9 of the 12 tasks studied. The absolute gains were 8.9% on commonsense reasoning, measured with the Stories Cloze Test, 5.7% on question answering, measured with RACE, and 1.5% on textual entailment, measured with MultiNLI ^[1].

OpenAI released the code and model on GitHub in the finetune-transformer-lm repository. That repository reproduces the ROCStories Cloze result of 86.5%. The repository is now archived ^[3].

Place in the GPT series

GPT-1 is the first model of the GPT series. It was succeeded by GPT-2 in February 2019 ^[2]. The series' main idea, pre-training a large language model on text and then adapting it, carried through every later generation. GPT-2 kept the same general approach and has its own entry.

For readers, GPT-1 is a useful reference point because it is small by later standards. Its 12 layers and 768-dimensional states are modest next to the models that followed, and its results are best read as early evidence for a method rather than as a finished product.