Mistral 7B

Mistral 7B is a language model with 7.3 billion parameters, released by Mistral AI on 27 September 2023 as the company's first model ^[1]. Mistral described it at launch as "the most powerful language model for its size to date" ^[1]. The comparison that mattered was against much larger networks: a model this small was meant to do the work of models several times its size.

Architecture

Two decisions about attention shape the model. It uses grouped-query attention (GQA), in which several query heads share one set of key and value heads ^[1]. Generating text means reading the cached keys and values again at every step, so keeping fewer of them reduces memory traffic and makes inference faster ^[1]. It also uses sliding-window attention (SWA): each layer attends only to the previous 4,096 hidden states rather than to the whole sequence ^[1]. The work done per layer is therefore bounded by the window instead of growing with the full input.

The two choices meet in the key-value cache. A rolling-buffer cache sized to the window overwrites entries that fall out of the window instead of storing them for the entire sequence. Mistral reported that at a sequence length of 8,192 tokens this halves the cache memory required for inference without affecting model quality ^[1].

Reported performance

Mistral's evaluation reported that Mistral 7B outperforms Llama 2 13B on all of the benchmarks it measured and Llama 1 34B on many of them ^[1]. On code it approaches CodeLlama 7B while remaining good at English-language tasks ^[1]. These are the company's own comparisons, published alongside the model.

Release terms

The weights were released under the Apache 2.0 license with no usage restrictions ^[1]. They were offered as a direct download and on Hugging Face, together with a reference implementation for deployment via vLLM and SkyPilot ^[1]. A permissive license plus a worked example of serving the model lowered the amount of work needed to put it into use.

The Instruct version

Mistral 7B Instruct was fine-tuned on public instruction datasets ^[1]. According to Mistral, it outperformed all 7B models on MT-Bench, a benchmark built from multi-turn questions used to compare chat models ^[1]. The model shipped with no built-in moderation ^[1].

The paper

A paper titled "Mistral 7B" was posted to arXiv on 10 October 2023 ^[2]. It lists 18 authors, among them Albert Q. Jiang, Arthur Mensch, Guillaume Lample and Timothée Lacroix, with Jiang listed first and Mensch third ^[2].

Place in the line

Mistral 7B was the first model Mistral AI published, and it set the pattern the company's later releases followed: open weights, a size chosen to fit ordinary hardware, and architectural tricks aimed at cheap inference ^[1]. The model also lived on as a hosted service; the API versions open-mistral-7b v0.1 through v0.3 were deprecated on 30 November 2024 and retired on 30 March 2025 ^[3].