EleutherAI releases GPT-Neo

Background

A large language model is a neural network trained on large amounts of text to predict what comes next in a passage; once trained, it can be prompted to continue, answer or rewrite text. Access to such models is a separate question from their existence. OpenAI's GPT-3 was reachable only through an API, meaning users sent requests to OpenAI's servers rather than obtaining the model itself ^[2]. For researchers who wanted to study a model's behaviour from the inside, or to run it on their own hardware, that arrangement set a limit that no amount of clever prompting could remove.

The release

On 21 March 2021, EleutherAI released the GPT-Neo models ^[1]. They came in three sizes: 125 million, 1.3 billion and 2.7 billion parameters ^[1]. Parameters are the learned numbers inside the network — roughly, the knobs adjusted during training — so the three versions represented a small model, a mid-sized one and a considerably larger one, published together so that users could pick the one that fit their hardware and task.

The decisive difference from GPT-3 was distribution: the GPT-Neo weights were freely downloadable ^[2]. Weights are the trained parameter values themselves, the file that turns an architecture into a working model. Publishing them meant that anyone who could download a file could hold the model rather than borrow it.

How they were trained

The models were trained on The Pile, EleutherAI's own dataset ^[2]. The model card states that The Pile was created by EleutherAI for the specific purpose of training these models ^[3], so data and model were designed together rather than the model being fitted to a corpus assembled by someone else.

The compute came from Google TPUs made available through the TPU Research Cloud ^[1]. TPUs are chips built for the matrix arithmetic that dominates neural network training, and cloud access to them let the project train at a scale it could not have reached on its own machines.

The model card gives a concrete account of the training run for the 1.3-billion-parameter version: 380 billion tokens over 362,000 steps ^[3]. Tokens are the chunks of text — words or word fragments — that the model reads and predicts; a step is one update of the parameters on one batch of that text. The ratio between the two numbers describes a long run over a very large amount of text, in which each batch is seen once and the parameters are nudged repeatedly towards better predictions.

Why it mattered

The contrast with GPT-3 was explicit in how the release was presented: weights were freely downloadable, unlike an API-only model ^[2]. That changed what outside researchers could do. A downloadable model can be inspected, run repeatedly on the same inputs, modified, and compared against other models under identical conditions; an API-only model can only be queried, subject to the provider's terms and availability.

Just as important was the pairing of a model with a documented dataset created for it ^[3]. A training corpus built by the same group that trained the model, together with a model card that describes the run, is what allows later work to say precisely what a result was built on. The three published sizes also gave the research community a ladder: the same recipe at different scales, small enough for modest hardware and large enough to be interesting. GPT-Neo is best understood not as a single model but as a package — weights, data and documentation released together on 21 March 2021 ^[1][2].