GPT-3

What it is

GPT-3 is an autoregressive language model with 175 billion parameters, described in OpenAI's paper "Language Models are Few-Shot Learners" by Tom B. Brown and 30 co-authors. The paper was posted to arXiv on 28 May 2020. At 175 billion parameters, it was 10 times larger than any previous non-sparse language model ^[1]. Autoregressive means the model generates text one token at a time, each token conditioned on the ones before it.

Origins

GPT-3 uses the same model and architecture as GPT-2, scaled up. It has 96 layers, a model dimension of 12,288, 96 attention heads and a context window of 2,048 tokens. The paper trained eight models, ranging from 125 million parameters to 175 billion ^[1].

All of the models were trained on 300 billion tokens. The training mix was weighted: 60% filtered Common Crawl, 22% WebText2, 8% Books1, 8% Books2 and 3% Wikipedia ^[1]. Common Crawl is a large archive of web pages, and the weighting shows that the data was drawn from several sources, not from a single body of text.

The paper's contribution statement says that Dario Amodei designed and led the research. Jared Kaplan and Sam McCandlish predicted that a giant language model should show continued gains. Alec Radford originally demonstrated that few-shot learning occurs in language models ^[1].

GPT-3 is the successor of GPT-2, and it was followed by the GPT-3.5 series. That series was the first model family behind ChatGPT, launched in November 2022, and was followed by GPT-4 in March 2023 ^[7].

Main features

The key result was few-shot, one-shot and zero-shot learning. In these settings the model performs a new task from a few examples, a single example, or only an instruction in the prompt, with no gradient updates. A gradient update is the step in which a standard model's weights are changed to fit new training data, so the absence of updates means the model's behaviour changed only through the text it was given ^[1].

As an example, the paper reports 64.3% accuracy zero-shot and 71.2% few-shot on TriviaQA, a trivia question-answering dataset ^[1].

The paper also tested human judgement. Human evaluators distinguished roughly 500-word news articles written by the 175-billion-parameter model from human-written articles with only about 52% accuracy, which is close to chance ^[1].

Access and licensing

Access was through OpenAI's API, announced on 11 June 2020. The waitlist for the API was removed on 18 November 2021 ^[5][6].

On 22 September 2020 Microsoft announced an exclusive license to GPT-3. Others could use the public API, but only Microsoft had access to the underlying model ^[2][6].

Recognition and retirement

The paper received a NeurIPS 2020 Best Paper Award ^[3].

The original GPT-3 base models in the API, named ada, babbage, curie and davinci, were shut down on 4 January 2024. Their replacements, davinci-002 and babbage-002, were scheduled to shut down on 28 September 2026 ^[4].

Place in the development of language models

GPT-3 showed that scale alone could give a single model a broad range of abilities that previously required separate task-specific training. Its prompt-based approach, in which the task is described in the input rather than in a retrained model, became a common way of using language models. Later models, including the GPT-3.5 and GPT-4 families, built on the same family of ideas.