StarCoder2

What it is

StarCoder2 is a family of open code models — language models trained mainly on source code, used to write, complete and edit programs — released on 28 February 2024 by BigCode, a collaboration between Hugging Face and ServiceNow, working together with NVIDIA^[1]. It shipped in three sizes at once: 3B, 7B and 15B parameters^[1].

Where it came from

StarCoder2 is the successor of StarCoder, released in May 2023, and was trained on roughly four times as much data as that earlier model^[2]. The jump was made possible in part by the dataset: The Stack v2, built from the Software Heritage source-code archive^[1]. The full dataset is 67.5 TB, against 6.4 TB for the first version of The Stack, and works out to about 900 billion training tokens where the earlier collection had about 200 billion^[1].

How it was built

The three sizes were split across the partner organisations: the 3B model was trained by ServiceNow, the 7B by Hugging Face, and the 15B by NVIDIA using its NeMo framework^[1]. All three share the same setup. The context window is 16,384 tokens, but attention is computed inside a 4,096-token sliding window rather than over the whole context at once^[1]. The models use grouped-query attention, a variant that shares key and value projections across groups of attention heads, and they are trained with a fill-in-the-middle objective: during training the model is asked to fill a gap in the middle of a piece of code, not only to continue from the end^[1]. That objective matches how code is actually edited, where new lines get inserted between existing ones.

Training data differed by size. The 3B model saw more than 3 trillion tokens drawn from 17 programming languages, the 7B more than 3.5 trillion tokens across the same 17 languages, and the 15B more than 4 trillion tokens spanning over 600 programming languages^[1]. The 3B model's data came from The Stack v2 restricted to those 17 languages, while the 15B flagship drew on the 600+ language portion^[3].

Results

Hugging Face described StarCoder2-15B as the best model in its size class and said it matches models of 33B parameters and larger on many evaluations; it also said the small StarCoder2-3B matches the original 15B StarCoder^[1]. Those are the claims made by the releasing organisation at launch; the interesting part of the comparison is that the smallest model of the new family was presented as catching up with the previous generation's largest one.

What was released

Alongside the announcement, BigCode released all of the StarCoder2 models, the Stack v2 dataset, and the code used for data processing and training^[3]. The models are downloadable from the Hugging Face Hub and are released under the BigCode OpenRAIL-M v1 licence^[1].

Where it sits

In the line of open code models, StarCoder2 is notable for two things. The first is scale: between StarCoder and StarCoder2 the training data grew about four times, and the context window and the fill-in-the-middle objective were set up around the way code is read and edited^[1][2]. The second is what was published. Putting out the models alone lets others use them; putting out the dataset and the processing and training code as well lets other teams reuse and adapt the pipeline that produced them^[3]. The three-size design also means the family spans from a small model meant to run cheaply to a flagship trained by NVIDIA, with each partner responsible for the size it trained^[1].