Multimodal models

What counts as multimodal

A multimodal model handles more than one kind of input or output — images alongside text, for example — in a single model, rather than keeping a vision system and a language system in separate boxes^[1][2]. The difference is not merely that two file formats are accepted. What the model reads can change how it processes what it sees, and the other way round, because both are handled by one set of weights.

How such a model learns

The clearest early recipe came from CLIP. Radford et al. trained it to predict which caption goes with which image, using a dataset of 400 million image-text pairs collected from the internet^[1]. Nothing in that objective is specific to any downstream task: the model only has to decide whether a piece of text and a picture belong together. That simple pre-training task proved an efficient and scalable way to learn state-of-the-art image representations from scratch^[1].

A small illustration of what the task demands. Suppose a batch contains a photo of a dog running on grass and a photo of a parked car, together with the sentences "a dog running on grass" and "a car parked on the street". To do well, the model must pull each picture towards its own sentence and push it away from the other. Surviving that test means noticing objects, settings and something about what is happening — exactly the information the captions tend to describe.

After pre-training, natural language can reference the visual concepts the model has learned, and that is what enables zero-shot transfer to downstream tasks^[1]. A new task can be named in words instead of being taught through labelled examples.

Two milestones

CLIP was published by Radford et al. in February 2021 and benchmarked on over 30 existing computer vision datasets, spanning OCR, action recognition in video, geo-localisation and fine-grained object classification^[1]. On ImageNet it matched the accuracy of the original ResNet-50 in the zero-shot setting, without using any of the 1.28 million training examples that ResNet-50 was trained on^[1].

Flamingo, published by Alayrac et al. in April 2022, is a family of visual language models designed to adapt to novel tasks from only a handful of annotated examples^[2]. It bridges pretrained vision-only and language-only models, and it handles sequences of arbitrarily interleaved visual and textual data, ingesting images or videos as inputs^[2]. Because it can be trained on large multimodal web corpora in which text and images are interleaved, it acquires in-context few-shot learning capability^[2]. On numerous benchmarks a single Flamingo model outperformed models fine-tuned on thousands of times more task-specific data, simply by being prompted with task examples^[2].

Two ways of putting the pieces together

The two systems illustrate two routes into the same idea. CLIP uses language as supervision for vision: the text is a teaching signal during training, and afterwards it becomes a way of addressing what the model has learned^[1]. Flamingo keeps a pretrained language model and a pretrained vision model and connects them, so that pictures and text can arrive together in one stream and the task itself can be described in the prompt^[2]. The first route produces a model that recognises; the second produces one that can be handed a few examples and asked to continue.

Where this matters

Because the interface is language, a single model can be pointed at many different jobs instead of one. CLIP's evaluation reached across reading text in images, recognising actions in video, locating where a photograph was taken and telling similar objects apart^[1]. Flamingo's few-shot setting means a task can be specified by showing examples in the prompt rather than by collecting a large annotated dataset and fine-tuning^[2]. That changes what it costs to try a new task: the work moves from training to describing.

How to read the results

These numbers are comparisons against named baselines, and they should be read that way. CLIP's zero-shot ImageNet result is stated against the original ResNet-50, a model trained on 1.28 million labelled examples^[1]; the claim is that the two reach similar accuracy by different means. Flamingo's comparison is against models fine-tuned on far more task-specific data, with Flamingo given only a handful of examples in its prompt^[2]. The benchmarks used cover particular families of tasks — the over 30 datasets listed for CLIP, the numerous benchmarks reported for Flamingo — so the results describe performance on those tasks, measured in that way^[1][2].