GPT-Neo

What it is

GPT-Neo is a family of open text-generation models released by EleutherAI on 21 March 2021, built as a replication of the GPT-3 architecture ^[1][5]. The first release came in three sizes: 125M, 1.3B and 2.7B parameters ^[1]. A parameter count is a rough measure of model capacity — each parameter is a number the model adjusts during training, so the 2.7B model has roughly twenty times as many adjustable numbers as the 125M one. The models are documented as English-language and are released under the MIT license ^[5].

How it was trained

The training dataset was The Pile, and the 1.3B model was trained for 380 billion tokens over 362,000 steps ^[2][3]. A token is a chunk of text, often a word or part of a word, so 380 billion of them is far more text than any person could read. Weights and configuration files were made freely downloadable from the-eye.eu ^[2][3]. According to the model card, GPT-Neo 1.3B was trained as a masked autoregressive language model with cross-entropy loss ^[5]. Autoregressive means the model is trained to predict the next token given the tokens before it; cross-entropy loss is the standard way of scoring how far the predicted probability distribution sits from the token that actually came next, and training steadily pushes that score down.

How it was built

The codebase was written in Mesh TensorFlow and targeted Google TPUs, which EleutherAI accessed through the TPU Research Cloud ^[1][4]. Writing a training stack for tensor processors rather than GPUs shapes how the work is organised: Mesh TensorFlow describes a computation as a mesh of operations spread across many chips, and the same description can be run at different sizes. That choice of backend is also why the project later needed a separate path for GPU users.

Release and distribution

The models are credited to Sid Black, Stella Biderman, Leo Gao, Phil Wang and Connor Leahy ^[3]. They were added to the Hugging Face Transformers library under the name GPTNeo ^[3], which meant users could load them through the same interface as other Transformer models instead of wiring up EleutherAI's own code. The MIT license places few restrictions on reuse, so the weights could be downloaded, modified and redistributed ^[3][5]. Because the weights and configurations were freely available ^[2][3] and the license permitted reuse ^[3][5], researchers could run, inspect and adapt the models themselves rather than only querying a hosted service.

What came after

The original repository was archived read-only on 25 February 2022, with development moving to the GPU-based GPT-NeoX codebase ^[3][4]. GPT-Neo itself was followed by GPT-J-6B in June 2021 and GPT-NeoX-20B in February 2022 ^[1], a progression that kept enlarging the scale of the models EleutherAI trained and released. GPT-Neo therefore sits at the start of that line: the point where the group moved from describing an open replication of GPT-3 to shipping actual weights that anyone could download and run.