Mistral AI releases Pixtral 12B, its first multimodal model

Background

A multimodal model takes more than one kind of input — usually an image alongside text — and answers in text. Two things have to line up for that to work: a way of turning a picture into a sequence of numbers the language model can read, and a language model strong enough to reason over what it is given.

What was announced

On 17 September 2024, Mistral AI announced Pixtral 12B, a model that accepts images and text as input.^[1] Its weights were released under the Apache 2.0 licence, which allows use and modification, including commercially.^[1] Mistral's changelog lists the release under the name pixtral-12b-2409.^[2]

How it is put together

Pixtral has two parts: a vision encoder that tokenizes images, and a multimodal transformer decoder.^[1] The encoder has 400 million parameters and was trained from scratch rather than taken from an existing vision model; it is paired with a 12-billion-parameter decoder based on Mistral NeMo.^[1]

Splitting the work this way has a purpose. The encoder cuts an image into patches and turns each patch into a vector, so the decoder can handle a picture much as it handles a sequence of words. Reading the small print on a scanned page, or working out how two objects in a photo relate, is then left to the decoder, which is the half doing the reasoning.

A token, in a language model, is the smallest unit being processed — roughly a word or part of a word. Once an image is cut into patches, each patch becomes a token too, and the picture therefore occupies positions in the model's context. A larger image, or finer slicing, takes up more of that space, which is the trade-off such models make between detail and how much can be handled at once.

Training the vision encoder from scratch, rather than starting from a finished one, means its behaviour is fitted to this particular pairing instead of to whatever task it was originally built for. That is a design choice about how the two halves learn to work together.

Reported results

Mistral reported 52.5% on MMMU with chain-of-thought prompting — asking the model to work through its reasoning step by step before answering — and 90.7 on DocVQA.^[1] The two benchmarks probe different things: MMMU covers knowledge-heavy questions across several academic subjects and image types, so it rewards reasoning; DocVQA asks questions about document images, so it rewards reading fine detail accurately. One looks at reasoning, the other at detail.

The same day

Mistral also made Mistral Small v24.09 available on its API on the same day.^[2] The two announcements are different in kind: one is an open-weight multimodal model, the other is a model reached through the API.

Later status

Mistral's Pixtral 12B page now states that the model is no longer maintained.^[1] Not being maintained does not mean the model disappears — the weights are still there and can still be run — but it stops receiving fixes and updates.