StarCoder

What it is StarCoder is an open-access language model for code. It was released on 4 May 2023 by BigCode, an open scientific collaboration led jointly by Hugging Face and ServiceNow.^[1] It has about 15.5 billion parameters and a context length of more than 8,000 tokens, which is the amount of text it can take into account at one time. It also has fill-in-the-middle, or infilling, ability, meaning it can fill a gap in the middle of a piece of code using the code before and after the gap. It uses multi-query attention, a technique that speeds up generation at inference time.^[1][2] ## Origins BigCode is a collaboration that brings together researchers from different institutions around a shared goal of building open code models. Hugging Face and ServiceNow led it jointly. The StarCoder models were released as the main output of that work, together with a technical paper that was posted to arXiv on 9 May 2023.^[2] The base model, StarCoderBase, was trained on about 1 trillion tokens from The Stack, which covers more than 80 programming languages. Its training data also included GitHub issues, Git commits and Jupyter notebooks.^[1][2] The Stack is a large collection of source code, and a Jupyter notebook is a document that mixes code, text and results. StarCoder itself is StarCoderBase further fine-tuned on 35 billion Python tokens.^[1] Fine-tuning here means extra training on a narrower set of examples, which in this case focused the model on Python code. ## Results On the HumanEval benchmark, a test of whether a model can write correct functions from descriptions, StarCoder reached a pass@1 score of 33.6%. StarCoderBase reached 30.4%.^[1] Pass@1 is the share of problems solved on the first attempt. With a prompt that framed the model as a technical assistant, StarCoder reached 40.8%.^[1] The prompt change shows that the way a question is posed can affect the measured result. ## Licence and data practices The models were released under the BigCode OpenRAIL-M v1 license. It allows commercial use, with use restrictions. Data processing and training code were released under Apache 2.0.^[1] The project also gave developers a way to opt out their code from The Stack. A PII-redaction pipeline, built with Toloka, removed names, passwords and email addresses from the training data.^[1] PII stands for personally identifiable information, and redacting it means replacing or removing such details from the data before training. ## Tools and successors Hugging Face shipped a VS Code extension for coding with StarCoder.^[1] VS Code is a widely used code editor, and an extension adds features to it. StarCoder was succeeded by StarCoder2 in February 2024.^[1] ## Impact StarCoder is an early example of an open model built specifically for programming tasks. Its data practices, including the opt-out process and PII redaction, are part of the record of how the project handled training data. Readers who want to use it should read the license and the paper directly, because the terms and the results are documented there.