Mistral AI releases Codestral Mamba and Mathstral

Background

A general-purpose language model is trained on everything at once. A specialised model is trained mainly on one kind of material — source code, or mathematics — and gives up breadth in exchange for accuracy on a narrow task. Because such models are usually much smaller than flagship systems, they can be run and inspected by people outside the company that built them. 7B refers to seven billion parameters, a size that fits on a single machine with a capable GPU.

Almost all models of this generation are built on the Transformer, whose self-attention compares every token in a sequence with every other token. That design is effective but its cost grows quickly with sequence length, which matters when the input is a whole file or a whole repository of code.

What happened

On 16 July 2024 Mistral AI released two specialised 7B open models on the same day, one aimed at code and one at mathematics ^[1][2]. They were published as a pair, but they differ in almost everything except size and domain focus: one changes the underlying architecture, the other changes the training data and the base model it starts from.

Codestral Mamba

Codestral Mamba does not use a Transformer. It is built on the Mamba architecture, and it was tested up to 256K tokens of context ^[1]. Long context is useful for code, where a model often has to keep a large amount of surrounding material in view while producing a single function or a patch. The model was released under the Apache 2.0 licence, which permits use and modification with few restrictions ^[1]. Albert Gu and Tri Dao helped design it ^[1].

Mamba belongs to the family of state-space models. Instead of storing a full table of comparisons between all pairs of tokens, such a model compresses what it has read so far into a running internal state and updates that state as each new token arrives; in principle this makes the cost grow linearly with sequence length rather than quadratically. Whether such an architecture matches attention on hard tasks is exactly the kind of question a released model lets outsiders test.

The model card reports 75.0% on HumanEval ^[3]. HumanEval is a benchmark made of short programming problems in which the model must complete a function from its signature and docstring, and the reported figure measures how often the produced code passes the accompanying tests.

Mathstral

Mathstral takes a different route: it is based on Mistral 7B and was built together with Project Numina ^[2]. It scored 56.6% on MATH ^[2]. MATH is a benchmark of competition-style problems that require multi-step reasoning and a written solution, so a result on it reflects the model's ability to carry a chain of argument through to a final answer rather than to recall a fact.

Its model card reports 77.1% on GSM8K in the 8-shot setting ^[4]. GSM8K consists of grade-school arithmetic word problems; 8-shot means that eight worked examples are placed in the prompt before the question, so the model is shown the expected format and reasoning style rather than being fine-tuned for it.

What it means

The release shows two ways of making a small model better at one thing. One is architectural: replace the component that everyone else uses and see whether a different design handles long inputs more cheaply. The other is data-driven and collaborative: start from an existing base model and work with an outside project that specialises in the domain. Publishing both as open 7B models put them within reach of researchers and developers who cannot train such systems themselves, and gave them a common scale at which the two approaches could be compared.

Mistral AI framed the pair as specialised models rather than general assistants, and both are aimed at settings where the task is narrow and verifiable — code that either runs or does not, an answer that is either right or wrong.