GPT-NeoX-20B

What it is

GPT-NeoX-20B is an open language model built by EleutherAI and released in February 2022^[1]. The name states its size, and the exact figures are public: 20,554,567,680 parameters arranged in 44 layers, a model dimension of 6,144, 64 attention heads, a context of up to 2,048 tokens, and rotary position embeddings^[3].

It is an autoregressive model, which means it is trained to predict the next token given everything that comes before it. Generating text is that single operation repeated: the model produces a probability distribution over the next token, one is chosen and appended to the context, and the process runs again. Models of this shape can be pointed at tasks they were never explicitly trained on by writing the task into the prompt.

Where it came from

EleutherAI announced the model on 2 February 2022 and released the weights on 9 February 2022 — a seven-day delay the organisation described as a thank-you to the donors who provided compute^[1]. It followed GPT-J-6B, released in June 2021, in EleutherAI's line of open GPT-style models^[1].

Training used The Pile, in a form that was not deduplicated, and ran on EleutherAI's own GPT-NeoX framework on GPUs provided by CoreWeave^[1][3].

Results and access

The announcement reported zero-shot scores of 72.0% on LAMBADA, against 75.2% for GPT-3 DaVinci, 66.1% on WinoGrande and 53.5% on HellaSwag^[1]. EleutherAI described the model as, to its knowledge, the largest publicly accessible pretrained general-purpose autoregressive language model at the time^[1].

Access was broad by design. The weights were released under the Apache 2.0 licence, could be downloaded from The Eye, and were also offered as a hosted service through GooseAI^[1].

The technical report

A technical report appeared on arXiv on 14 April 2022, by Sid Black, Stella Biderman, Eric Hallahan and others, and was presented at an ACL 2022 workshop; it describes the model as a particularly strong few-shot reasoner^[2]. The paper states that, to the authors' knowledge, GPT-NeoX-20B was the largest dense autoregressive model with publicly available weights at the time of submission^[4]. The authors also report that the model gained far more from five-shot evaluation than similarly sized GPT-3 and FairSeq models did^[4]. Training and evaluation code were open-sourced together with the weights^[4].

Why it mattered

Putting the weights out means a model can be downloaded, inspected, run on one's own hardware and modified, rather than only called through someone else's service; the Apache 2.0 licence is a permissive open-source licence for reuse. The same release also made the surrounding machinery public: the code used to train and evaluate the model ships with it, so the reported numbers can be checked against the recipe that produced them^[4].

The numbers themselves tell a modest story. On LAMBADA the model sat behind GPT-3 DaVinci, 72.0% against 75.2%, though in the same range^[1]. Its distinguishing behaviour, as reported by its authors, showed up in few-shot settings, where it improved more from five examples than comparably sized GPT-3 and FairSeq models^[4].

In EleutherAI's line of open GPT-style models, GPT-NeoX-20B is the step after GPT-J-6B, raising the scale of what the group published openly by roughly a factor of three^[1]. It is a checkpoint in that line rather than an endpoint, and much of its value comes from being something other people can build on rather than something they can only query.