OpenAI releases the full 1.5-billion-parameter GPT-2

Background

During 2019 OpenAI released GPT-2 in stages, and each earlier release was smaller than the full model. The stages were a 124M model in February, a 355M model in May and a 774M model in August.^[1]

A staged release means publishing parts of a project in sequence rather than all at once. Each stage gives outside observers a chance to see how the previous one was handled before the next is taken.^[1] Researchers studying the model could compare its outputs with those of the earlier, smaller versions, which is one reason the staged approach gave the community a series of models to examine.

What happened

On 5 November 2019 OpenAI released the largest version of GPT-2, with 1.5 billion parameters, along with code and model weights. It was the final stage of the staged release.^[1]

OpenAI said the release was meant in part to facilitate detection of text generated by GPT-2 models.^[1] It also noted that larger language models had been released by others since August, but said it had kept to its original staged-release plan, to give the community a test case of a full staged release.^[1]

Key details

The model card describes the 1.5B model as the fourth and largest GPT-2 version. It is dated February 2019 and was trained on data that cuts off at the end of 2017. The WebText training set came from the text of 45 million links posted on Reddit.^[2]

The model card warns that the model cannot tell fact from fiction and that it reflects biases in its training data. It reports that OpenAI's fine-tuned detector reached about 95% accuracy, but says that no single detection method is sufficient.^[2]

OpenAI also released a dataset of GPT-2 outputs, called gpt-2-output-dataset, so that researchers could study the models' behaviour.^[3]

The gpt-2 code repository uses a Modified MIT license and is now archived. The staged-release report on arXiv was revised on 13 November 2019.^[3][4]

Impact

With this release the staged plan was complete. Its main legacy is the way it treated the release as something to study and document: the model, the model card and the dataset of outputs were published together, alongside a separate report on the social impacts of language models. The model card's warnings also set out a point that holds for any language model. Fluent output is not the same as accurate output, and a detector can be only part of the answer.

Readers of the model card should note that its warnings describe the model as it was released in 2019, and that later systems may behave differently.^[2]

Taken together, the three earlier stages and the final release show a pattern of releasing a model in increments and documenting each step. That pattern can be judged on its own terms, separate from the model's capabilities.