EleutherAI releases GPT-NeoX-20B

Background

Training a 20-billion-parameter model needed substantial compute. For this release, the GPUs were supplied by CoreWeave, a cloud provider. ^[1]

What happened

EleutherAI announced GPT-NeoX-20B on 2 February 2022. The weights were released on 9 February. The model has 20 billion parameters. ^[1]

The group said that, to its knowledge, the model was the largest publicly accessible pretrained general-purpose autoregressive model. ^[1]

Key details

Training used GPUs from CoreWeave and took about a year of effort. The work was slowed by shipping delays linked to a chip shortage. ^[1]

The weights were released under Apache 2.0, and the model was served through GooseAI, a hosted service. ^[1]

EleutherAI cautioned that the model was a research artifact. It was not recommended for production use without careful consideration. ^[1]

Impact

The release gave researchers and developers a large open model they could study and run themselves, rather than only through an API. The group's caution about production use is part of the record: an open model of this size was offered as a research tool, with its limits stated plainly. ^[1]

What a research artifact means

A research artifact is a model released so that others can study it, test its behaviour and build on it. It is not a finished product, and its outputs can be wrong, biased or unsuitable for a given purpose. EleutherAI's caution was a reminder that publishing weights openly does not by itself make a model ready for production use. ^[1]

Access

An open release lets people download the weights and run the model on their own hardware. A hosted service lets them use the model through an interface without managing it themselves. This model was available both ways. ^[1] The announcement and the weight release were therefore a week apart, and readers who want the training details should read the group's own announcement, which is the source for this entry. ^[1]