EleutherAI releases GPT-J-6B

Background

A language model of this kind is trained on a single task: given a stretch of text, predict what comes next. Its parameters are the numbers learned while doing that, and a model with billions of them needs an enormous amount of text and a great deal of computation to train. Once training is finished, a second question follows: who gets to use the result. A model can be kept behind a hosted interface that answers requests, or its weights — the trained parameter files — can be published so that anyone can download and run them. GPT-J-6B belongs to the second category.

What was released

EleutherAI released GPT-J-6B in June 2021 as an open model with 6 billion parameters ^[1]. Aran Komatsuzaki proposed the project and wrote the announcement, and Ben Wang wrote the JAX code ^[1]. The weights were released under Apache 2.0 ^[2], a permissive licence that allows the model to be reused and modified.

The choice of JAX, a numerical computing framework, matters for the people who later want to read or adapt the training and inference code rather than only download the finished weights.

How it was trained

Training ran on The Pile, a text corpus, using Google Cloud TPUs — specifically a TPU v3-256 pod — and processed 402 billion tokens ^[2]. A token is the unit a model actually reads; it is roughly a word or a piece of a word, so the token count describes how much text passed through the model during training.

The architecture is described in the model card: 28 layers, each split into 16 attention heads of 256 dimensions, with a context of 2,048 tokens ^[3]. Attention heads are the parallel paths through which the model relates different positions in the text to one another; the context length sets how much text it can take in at once, and anything beyond that window does not enter the computation. The model reuses the 50,257-token byte-pair encoding (BPE) vocabulary of GPT-2 and GPT-3 ^[3]. A vocabulary fixes how raw text is chopped into units, so sharing one with the earlier models means the same string is split into the same pieces.

Results

The announcement stated that the model performed close to the 6.7B GPT-3 (Curie) on zero-shot tasks ^[1]. Zero-shot means the model is given a task description and an input with no worked examples and is expected to answer directly. The comparison is the part that drew attention: a model whose weights are public was being measured against a comparable model that was not.

Why it matters

Because the weights are downloadable under a permissive licence, other researchers can run the model themselves, inspect it, and build on it without going through a hosted service ^[2]. That changes what kind of work is possible: results can be reproduced and probed on local hardware, and the model can be adapted to new uses under the terms of the licence. For a field where the largest models were mostly reachable only as a service, publishing a 6-billion-parameter model as files was a concrete alternative.

It also left a record. The announcement, the licence and the model card document who wrote which part of the code, how much data was used, what hardware ran the training, and what the architecture looked like down to the number of heads and the size of the vocabulary ^[1][2][3]. That level of detail is what makes a released model something others can check rather than simply use.