Mistral AI releases Mistral 7B, its first open-weight model

Background

The company had been founded less than five months earlier. ^[1][3] The weights were distributed as a direct download and on Hugging Face. ^[1]

Open-weight means that the trained parameters of a model are published, so anyone can download and run it, although the training data and code may not be released. Apache 2.0 is a permissive licence that allows use, modification and redistribution, including commercial use, as long as the licence and copyright notices are kept. A model's parameter count is a rough measure of its size. A 7-billion-parameter model is small compared with the largest systems, so it can be run on more modest hardware, which made it practical for individuals and small teams.

What happened

On 27 September 2023 Mistral AI released Mistral 7B, a language model with 7.3 billion parameters. The release came less than five months after the company was founded. ^[1][3] Mistral 7B Instruct, a version fine-tuned to follow instructions, was released at the same time. ^[1]

The weights were released under the Apache 2.0 licence and distributed as a direct download and on Hugging Face. ^[1]

Key details

Mistral said the model outperforms Llama 2 13B on all benchmarks and Llama 1 34B on many of them. ^[1] The model uses grouped-query attention, which speeds up generation, and sliding-window attention, in which each layer looks at the previous 4,096 hidden states. ^[1]

The technical paper followed on arXiv on 10 October 2023. ^[2] The paper describes the architecture of the model. ^[2]

Benchmarks are standard test sets used to compare models, such as multiple-choice exams or coding problems. Comparisons between models depend on the test set, the prompt format and how results were measured, so a developer's own claims are best checked against independent evaluations.

Impact

The release marked the first open-weight model from Mistral AI. ^[1] A model with an open licence and a small size can be run by individual developers, universities and companies on their own hardware. ^[1]