BigCode releases the StarCoder open code model

Background

A code model is a language model trained largely on source code, so that it can complete a function, write a test or work through an error message. Building one requires an enormous corpus of code, and that raises two practical questions alongside the modelling work: which code may legitimately be used, and what happens to personal information that ends up inside repositories. BigCode, a project led by Hugging Face and ServiceNow, worked on both questions and released the resulting models rather than keeping them closed^[1].

What was released

On 4 May 2023 BigCode released two models, StarCoder and StarCoderBase^[1]. Both have roughly 15.5 billion parameters and a context window of 8,000 tokens, which means a prompt can carry a long stretch of code and still leave room for the model's answer^[2]. The family is described as having a context length of over 8,000 tokens^[3].

How they were trained

StarCoderBase was trained on about 1 trillion tokens taken from more than 80 programming languages in The Stack^[1]. The underlying material came from permissively licensed GitHub data, and beyond ordinary source files it included Git commits, GitHub issues and Jupyter notebooks^[3]. Commits and issues are conversational in form, and notebooks mix code with prose and saved outputs, so this mixture pushes the model past the task of finishing an isolated function.

StarCoder itself is StarCoderBase fine-tuned on 35 billion Python tokens^[3]. The two-model arrangement separates a broad base from a version tuned toward the language that accounts for a large share of both training data and real requests.

Results

The models were evaluated on HumanEval, a benchmark in which the model must write a function from its docstring and the answer is checked by running unit tests. StarCoder scored 33.6% pass@1 there^[1]. Because pass@1 counts only the first attempt, the figure reflects how often the model's initial answer was correct rather than how often it succeeds after several tries. With a technical-assistant prompt, which frames the request as coming to a coding helper, the score rose to 40.8%^[1]. The distance between the two numbers is a reminder that a benchmark result depends on how the model is asked, not only on the model.

Licensing and data handling

StarCoder and StarCoderBase were released under the BigCode OpenRAIL-M v1 license, and the project set up an opt-out process through which code authors could exclude their work^[1]. Training data also went through PII redaction^[1], and the release shipped an improved PII redaction pipeline as well as a new attribution tracing tool^[3]. Attribution tracing matters for code models specifically: generated code that closely reproduces a training example raises questions of credit and license that ordinary text generation rarely faces.

Why it matters

The release put a 15.5-billion-parameter code model into public hands under a named license, together with documentation of what went into it and how it was cleaned^[1][2][3]. That combination — a capable model, a stated license, an opt-out route for authors and a way to trace outputs back toward their sources — made the project a reference point for later arguments about how code models should be built and released. The HumanEval figures also gave later work a concrete score to compare against^[1].