Ai2 releases Molmo open vision-language models

Background

A vision-language model is a system that reads an image together with a text request and answers in text. Training one well takes very large collections of images paired with language, and that training is expensive, so some teams take a shortcut: they train their model to imitate the answers of an already strong proprietary model. Ai2 said Molmo was trained without distilling proprietary vision-language models, unlike several strong open-weights models.^[2]

What was released

On 25 September 2024, Ai2 released Molmo, a family of open multimodal models.^[1] It is a family rather than a single model: the range runs from MolmoE-1B at the small end to Molmo-72B at the large end.^[1] The models were trained on Ai2's new PixMo image datasets.^[1] Offering a spread of sizes matters in practice, because the model that has to run on modest hardware and the model that has to answer the hardest questions are different engineering problems.

Pointing at what it sees

The capability Ai2 put forward is pointing. Molmo can point at things in images in response to text.^[1] Most image-capable chat models answer in words — they say that a cup is on the left. Pointing is a different kind of output: the model marks where in the image something is. Ai2 described the models as learning to point at what they perceive, and presented this as a basis for acting in physical and virtual environments.^[2] The reasoning behind that framing is easy to follow. An assistant that only describes a scene can advise a person, but a system that has to move a robot arm or click a button needs the location of an object, not a sentence about the object.

How it compared

Ai2 said that Molmo-72B ranked just behind GPT-4o in human preference.^[1] Human-preference evaluation means showing people the outputs of two models and asking which they prefer, then aggregating the votes into a ranking. A result of that kind is a statement about how outputs read to people, not a claim of superiority on every task, and this one is Ai2's own characterisation of its model.

Why it matters

Two things stand out in how Ai2 described the release. The first is provenance. Ai2 said Molmo was trained without distilling proprietary vision-language models, unlike several strong open-weights models.^[2] In other words, the training did not run through the outputs of a closed system, and the models were built on Ai2's own new PixMo image datasets.^[1]

The second is what the model is built to do. Pointing connects description to location, and Ai2 presented learning to point at what one perceives as a basis for acting in physical and virtual environments.^[2] A model that can indicate where something is in an image is doing part of the work that any system acting on the world around it would need to do: locating the thing it is about to touch, pick up, or click.

The family also spans a wide range of sizes, from MolmoE-1B to Molmo-72B.^[1] Ai2's own comparison placed the largest model just behind GPT-4o in human preference.^[1] Taken together, the release was presented as an open multimodal family whose training path and output format — pointing, not only prose — were both deliberate choices.