Mistral AI and NVIDIA release Mistral NeMo 12B

Background

A language model never sees text directly. It sees tokens: the chunks of characters that a tokenizer cuts text into before the model processes it. The tokenizer therefore shapes what a model can do. One trained mostly on English squeezes English into few tokens and breaks other languages into many, which costs both context space and compute. The context window, the number of tokens a model can hold at once, sets how much text it can read in a single pass. Quantisation, storing weights in a lower-precision format such as 8-bit floating point (FP8), cuts memory use and speeds up inference, but a model trained without regard for it can lose accuracy once quantised. Licence terms decide who can actually run the model, and packaging decides how much work that takes.

The release

On 18 July 2024, Mistral AI released Mistral NeMo, a 12B model built together with NVIDIA ^[1]. The model card describes the instruct version as trained jointly by the two companies ^[2].

Key details

Mistral NeMo supports up to 128K tokens of context ^[1]. That is the working space for a single conversation or document: prompts, retrieved passages, examples and the model's own output all draw on the same budget.

The model introduced a new tokenizer called Tekken, trained on more than 100 languages ^[1]. Because the tokenizer's training data determines how text is split, a tokenizer covering many languages keeps non-English text from being fragmented into unusually long token sequences.

Mistral NeMo was trained with quantisation awareness for FP8 inference ^[1]. Rather than quantising a finished model after the fact, the training accounts for the lower-precision arithmetic that inference will use, so the compressed version stays closer to the behaviour of the full-precision one.

The weights were released under the Apache 2.0 licence and packaged as an NVIDIA NIM microservice ^[1]. Apache 2.0 is a permissive open-source licence, and a NIM microservice is a pre-built container for serving a model, so the release covers both the raw weights and a ready-made way to deploy them.

On the model card, the instruct version reports 68.0% on MMLU in the 5-shot setting ^[2]. MMLU is a benchmark of multiple-choice questions across academic subjects, and 5-shot means the model is shown five worked examples per task before answering.

Why it matters

The release ties several design decisions together. A long context window is only useful if the tokenizer does not waste it, so the multilingual Tekken tokenizer and the 128K-token limit work as a pair ^[1]. Training for FP8 inference addresses the practical gap between a model as published and a model as served: most deployments run in reduced precision to keep costs down, and building that assumption into training makes the two versions less different ^[1].

Releasing weights under Apache 2.0 and shipping a NIM microservice address two different audiences: developers who want to inspect, fine-tune or self-host the model, and teams that want to run it without assembling their own serving stack ^[1]. The reported 68.0% on MMLU gives a single reference point for comparing the instruct version with other models of similar size ^[2].