Codestral Mamba

What it is

Codestral Mamba is a code model released by Mistral AI on 16 July 2024.^[1] It has 7,285,403,648 parameters and is instruction-tuned, meaning it was trained to follow instructions rather than only to continue a piece of text.^[1] What sets it apart from most models of its generation is not size but architecture: instead of a Transformer, it is built on the Mamba state-space architecture, which gives linear-time inference as the input grows.^[1] The model card for the released weights describes it as an open code model based on the Mamba2 architecture.^[3]

Why the architecture matters

A Transformer's self-attention compares every token with the tokens before it, so the computation needed to process a sequence grows roughly with the square of its length. That is why long inputs are expensive. Mamba belongs to a different family, state-space models. Instead of keeping the whole past available for comparison, such a model carries a compressed internal state forward and updates it token by token, in the manner of a recurrent network. The work of producing each new token stays roughly constant, so total cost grows linearly with sequence length rather than quadratically.^[1]

The trade-off is that a compressed state has to summarise the past, and summarising can lose detail. Whether such a model can still find a specific fact far back in a long input is an empirical question, which is why retrieval tests over long contexts matter for this design.^[1]

Where it came from

Mistral designed the model with help from Albert Gu and Tri Dao, the authors of Mamba.^[1] The choice to work with the architecture's originators, rather than reimplementing it from the paper alone, is part of how the release was positioned.

Codestral Mamba was released under the Apache 2.0 licence, with weights published on Hugging Face and API access under the name codestral-mamba-2407.^[1] Open weights mattered here: an unusual architecture is easier to study, benchmark and build on when outside researchers can run the model themselves.

Long context and reported scores

The model was tested on in-context retrieval up to 256K tokens.^[1] Mistral said Codestral Mamba performs on par with state-of-the-art Transformer-based code models.^[1][3]

The model card gives a table of results for the 7B model: 75.0% on HumanEval, 68.5% on MBPP and 58.8% on Spider.^[3] HumanEval and MBPP test whether a model can write short programs from a plain-language description, and Spider tests turning questions into database queries. In the same table, the larger Codestral (22B) scored 81.1% on HumanEval and 78.2% on MBPP.^[3] Read side by side, the numbers say something useful: the Mamba-based model is competitive enough to be taken seriously, while the bigger model still leads on the two code-generation benchmarks. The "on par" framing and the table pull in slightly different directions, and the table is the more concrete of the two.

Where it sits

Codestral Mamba is best understood as a deliberate experiment made public. Nearly every code model of its era descends from the Transformer, and the claimed advantage of Mamba — cheaper inference on long inputs — only matters if quality holds up on real tasks. Releasing weights under a permissive licence let others check that claim instead of taking it on trust, and the 256K-token retrieval test was aimed at exactly the weakness a compressed state might have.^[1]

Its life as a hosted service was short. The model was retired from the API on 6 June 2025.^[2] The weights, released under Apache 2.0, remain the durable part of the release.^[1] For anyone studying which architectures can carry a code model at usable quality, Codestral Mamba is a documented data point: a 7B open model built on Mamba2, tested out to long contexts, and scored on standard benchmarks next to a larger sibling.