GPT-J-6B

What GPT-J-6B is

GPT-J-6B is an open language model with roughly six billion parameters, released by EleutherAI in June 2021^[2][3]. Its weights are published under the Apache 2.0 license, which permits reuse, including commercial use^[1]. At the time of release, Wikipedia described it as "the largest open-source GPT-3-like model in the world"^[3].

A model of this kind is trained on one basic task: predict the next token — a word fragment — given the text so far. Doing that well across a large corpus turns out to require, and to produce, a broad range of language abilities, which is why such models can be put to translation, summarisation and question answering without being trained for those tasks specifically.

Where it came from

The announcement post by Aran Komatsuzaki is dated 4 June 2021, while Wikipedia gives 9 June 2021 for the release^[2][3]. The model is cited as the work of Ben Wang and Aran Komatsuzaki, and was trained with Ben Wang's Mesh Transformer JAX codebase^[1]. It is the successor of GPT-Neo and the predecessor of GPT-NeoX-20B, which followed in February 2022^[3].

Architecture

The model has 6,053,381,344 parameters arranged in 28 layers, with a model dimension of 4096 and a context window of 2,048 tokens^[1]. A parameter is a number learned during training; the model dimension is the width of the vector that represents each token as it passes through the network; the context window is how much text the model can see at once when predicting. The 28 layers are split into 16 attention heads of 256 dimensions each, and the feedforward dimension is 16,384^[4]. Attention heads are the parts of a transformer that let each token draw information from other tokens in the context.

Rather than the learned position vectors used in earlier models, GPT-J uses rotary position embeddings (RoPE)^[1]. RoPE encodes position by rotating each token's representation, so the relative distance between two tokens shows up as an angle of rotation; in GPT-J it is applied to 64 dimensions of each head^[4].

The tokenizer uses a 50,257-token vocabulary and is the same byte-pair encoding (BPE) tokenizer as GPT-2 and GPT-3^[4]. Byte-pair encoding splits text into subword units, which lets a fixed-size vocabulary cover arbitrary text, including words it has never seen.

Training

GPT-J-6B was trained for 402 billion tokens of The Pile over 383,500 steps on a TPU v3-256 pod, about five weeks of training in total^[1][2]. Training on far more tokens than the model has parameters is the standard way to get a small model to use its capacity well: each parameter is updated many times over, on many different pieces of text.

Results

On the benchmarks listed in its model card, GPT-J-6B lands close to GPT-3 Curie, a 6.7-billion-parameter model: LAMBADA accuracy 69.7% against 69.3%, Winogrande 65.3% against 65.6%, HellaSwag 66.1% against 68.5%, and PIQA 76.5% against 77.9%^[1]. LAMBADA tests whether a model can supply the final word of a passage, which usually requires following the wider context; HellaSwag tests commonsense completion of a sentence; PIQA tests physical commonsense question answering; Winogrande tests working out which entity a pronoun or phrase refers to.

The pattern across those four numbers is a model that is level with its comparison point on some tasks and slightly behind on others —close enough that the difference between the two is smaller than the gap between either and much larger models.

Why it matters

GPT-J-6B's significance lies in the combination of size, openness and timing. A freely licensed model whose published scores matched a mid-sized proprietary GPT-3 variant gave researchers and small teams a strong baseline they could download, inspect and fine-tune themselves, instead of paying for API access. The Apache 2.0 license removed most legal ambiguity about doing so^[1].

It also marks a point on a visible trajectory. Coming after GPT-Neo and before GPT-NeoX-20B^[3], it shows an open project scaling up its models step by step, each release trained on more data and more compute than the last. For a reader trying to understand how openly available language models developed, GPT-J-6B is one of the reference points: a six-billion-parameter model, trained for five weeks on a single large TPU pod, that could be downloaded and run by anyone^[1][2].