GPT-2

What it is

GPT-2 is a language model announced by OpenAI on 14 February 2019. It is described in the paper "Language Models are Unsupervised Multitask Learners", by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever ^[1].

The largest version has 1.5 billion parameters, listed as 1542M in the paper. Parameters are the numerical values a model adjusts during training, and more parameters generally means more capacity to learn patterns. The model uses a Transformer-based architecture, a vocabulary of 50,257 tokens and a context of 1,024 tokens, up from 512 in the original GPT ^[1].

Origins

GPT-2 is the successor of GPT-1, which was published in 2018. It is the predecessor of GPT-3, released in 2020, which kept the same model and architecture at more than 100 times the size ^[4].

Its training data came from WebText, a new dataset built from the text of pages linked from Reddit posts. It contains slightly over 8 million documents and 40 GB of text, and Wikipedia articles were removed. The model card says the data came from 45 million links and cuts off at the end of 2017 ^[1][6].

Main features

Without any task-specific training, a setting called zero-shot, GPT-2 achieved state-of-the-art results on 7 of 8 tested language-modelling datasets. It also reached 55 F1 on CoQA, a reading-comprehension dataset ^[1].

Staged release and risk

OpenAI initially withheld the full model. It cited the risk of malicious use, such as spam that evades filters or obscene and racist text. Press headlines described the model as "too dangerous" to release ^[4].

The release happened in stages. The 124M-parameter model came out in February 2019, the 355M model in May 2019, and the 774M model in August 2019. The full 1.5-billion-parameter model was released on 5 November 2019 ^[4][5].

OpenAI said the final release was meant in part to help detect GPT-2 text. Its model card reports about 95% accuracy for OpenAI's fine-tuned detector, but warns that no single detection method is sufficient ^[5][6]. OpenAI documented the staged release and its risk research in the report "Release Strategies and the Social Impacts of Language Models", dated August 2019 and revised in November 2019 ^[2].

Code and later status

The code and weights were published in the openai/gpt-2 GitHub repository under a modified MIT license. The repository was archived, meaning it became read-only, in April 2026 ^[3].

GPT-2 is a clear example of the GPT series' growth. It builds on the same Transformer foundation as GPT-1, but it was trained on the WebText dataset and released in several sizes. It also shows how the staged release became a point of public debate about how large language models should be shared.