BigCode and NVIDIA release StarCoder2 and The Stack v2

Background

BigCode is a collaboration between Hugging Face and ServiceNow, and NVIDIA took part in this release ^[1]. The project works on code models, which are language models trained on source code. Such models can help with tasks like completing a line of code, explaining a function or finding a bug. Their quality depends heavily on the code they learn from, so the choice of training data is as important as the design of the model itself.

An open release matters to developers because the model weights can be downloaded, inspected and run on their own machines. Users still need to follow the terms of the licence that comes with the weights.

What happened

On 28 February 2024 BigCode released StarCoder2 in three sizes: 3B, 7B and 15B parameters ^[1]. A parameter is an adjustable number inside a model that stores part of what the model has learned, so the B in each name stands for billions of such numbers. The release came with a new training set called The Stack v2 ^[1].

The three models were not trained by one group. ServiceNow trained the 3B model, Hugging Face trained the 7B model, and NVIDIA trained the 15B model using its NeMo framework ^[1]. The division of work shows how a shared open project can bring together a software company, a model hub and a chip maker, each taking responsibility for one part of the family.

Key details

The 15B model was trained on more than 4 trillion tokens covering more than 600 programming languages ^[1]. A token is a small piece of text, such as a word or part of a word, so 4 trillion tokens means the model read a very large body of code and text during training.

The Stack v2 is 67.5 TB and is built from the Software Heritage archive of source code ^[1]. Its size is the main reason the training set matters. A larger and more varied body of code gives a model more examples of how programs are written in different languages.

The context window grew to 16,384 tokens ^[1]. A context window is the amount of text a model can take in at one time. A larger window lets the model see more of a project, such as the functions around the place where a programmer is working.

The models were released under the BigCode OpenRAIL-M v1 licence ^[1]. OpenRAIL licences are open-access licences that add responsible-use conditions, so the licence text should be read before the models are used in a product.

Impact

For developers, the release gave a choice of sizes within one family and one training set, rather than a single model. A user with limited hardware could pick a smaller model, and a user who needed broader language coverage could pick the 15B model. The three sizes were trained by different partners, which is itself an example of how open AI projects are assembled.

The 15B model covered more than 600 programming languages, a breadth that the training set was built to support. For readers who follow AI, the event shows how the scale of the training data and the openness of the release have become central questions in code models.