Stability AI launches the Stable LM language models

What was released

On 19 April 2023 Stability AI released the first models of its Stable LM family: alpha versions with 3 billion and 7 billion parameters ^[1]. Larger versions with 15 billion and 65 billion parameters were announced as coming later ^[1].

Parameters are the numbers a neural network learns during training. Roughly speaking, the parameter count reflects a model's capacity: more parameters can hold more patterns, but they also cost more computation to train and to run.

Training data

The models were trained on a new dataset built on The Pile with 1.5 trillion tokens ^[1]. A token is a chunk of text — often a word or part of a word — so a token count is a way of measuring how much text a model has been trained on. Stability said the new dataset was three times larger than The Pile itself ^[2].

Base models and instruction-tuned models

Two sets were published. The base models were licensed CC BY-SA-4.0, which allows commercial and research use ^[1]. A share-alike clause of that kind means that modified versions distributed to others have to carry the same license.

The instruction-tuned versions were released for research use only ^[1]. They were fine-tuned on a combination of five open source conversational datasets: Alpaca, GPT4All, Dolly, ShareGPT and HH ^[2]. Instruction tuning means continuing to train a base model on examples of a request paired with a good answer, so that it learns to respond to requests rather than merely continue text. A base model told to summarise a paragraph may simply keep writing in the same style; a tuned model is more likely to produce an actual summary. The instruction-tuned Stable LM models carried the noncommercial CC BY-NC-SA 4.0 license, in line with Stanford's Alpaca license ^[2].

What the models could do

Stable LM models can generate both text and code ^[2]. Stability said they performed well on conversational and coding tasks despite having only 3 to 7 billion parameters, compared with GPT-3's 175 billion ^[2].

Release and follow-up

The models were published in a GitHub repository ^[2]. Stability promised a full technical report later, and announced a crowd-sourced RLHF program together with work with Open Assistant ^[2]. RLHF, reinforcement learning from human feedback, is a way of tuning a model in which human judgements about which of several outputs is better are used as a training signal, instead of relying only on fixed text.

Why it mattered

The release is a clear example of how open language models were packaged. Two differently licensed sets shipped together: a base model that could be used commercially under a share-alike license, and an instruction-tuned model that was restricted to research, with a license chosen to match the one used for Stanford's Alpaca, whose data was among the sources of its tuning mixture ^[1][2]. The models were distributed as downloadable files in a public repository, and the accompanying statements covered the training data, the intended uses of each version, and a promise of a technical report to come ^[1][2]. Stability also framed the result in terms of size: models of 3 to 7 billion parameters were said to do well on conversation and code next to a 175-billion-parameter model ^[2]. Announcing a crowd-sourced RLHF effort alongside the weights pointed at the next step for such models — improving their behaviour with human feedback rather than with more pre-training data alone ^[2].