OpenAI announces GPT-2 and holds back the full model

Background

A language model learns to predict the next word in a sequence of text. Training one on a large collection of text gives it broad knowledge of how language works, which can then be put to other uses. Parameters are the numerical values a model learns during training. A model with more parameters is larger and generally has more capacity to capture patterns.

Smaller and larger versions of a model differ mainly in their number of parameters, and the same training approach can be applied at several sizes.

What happened

On 14 February 2019 OpenAI announced GPT-2 in a blog post titled *Better language models and their implications*, and in a paper titled *Language Models are Unsupervised Multitask Learners*.^[1][2] The paper's authors were Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever.^[2]

Key details

The paper trained four model sizes, listed as 117M, 345M, 762M and 1542M parameters. The largest, which carries the GPT-2 name, was a 1.5-billion-parameter Transformer with 48 layers. OpenAI later said these original counts were wrong and renamed the sizes 124M, 355M, 774M and 1.5B.^[2][5]

The training data was WebText, a new web scrape built from outbound links from Reddit that had received at least 3 karma. It contained slightly over 8 million documents and 40 GB of text, with Wikipedia pages removed.^[2]

Without task-specific training, GPT-2 achieved state-of-the-art results on 7 of 8 tested language-modeling datasets in a zero-shot setting, meaning it was tested without any examples or training for the task. On the CoQA reading-comprehension dataset it reached 55 F1 without using its more than 127,000 training examples.^[2] OpenAI said the model generated coherent paragraphs and performed rudimentary reading comprehension, machine translation, question answering and summarization, all without task-specific training.^[1]

OpenAI did not release the full model at first. It released a 124M model in February 2019, a 355M model in May and a 774M model on 20 August 2019, in what it called a staged release.^[4]

OpenAI documented the approach in the report *Release Strategies and the Social Impacts of Language Models*, by Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell and others. It was first posted to arXiv on 24 August 2019.^[4]

Impact

The staged release made the publication process part of the story: the choice was explained in a public report, not only in the announcement. The report's existence means the release decision itself can be studied. A general limitation of models like this also applies. Fluent output and accurate output are different things, and anyone reading generated text should keep that in mind.

The model's results were reported in a zero-shot setting, which describes testing without any examples or training for the task. Such results depend on the benchmarks used to measure them, so they should be read as performance on those tests.^[2]