Flamingo

A model that reads images and text together

Flamingo is a visual language model from DeepMind, announced on 28 April 2022^[1]. Its input is a prompt in which images, videos and text are mixed together; its output is associated language^[1]. The aim is not one fixed task but a single model that can be pointed at many tasks through the prompt it is given.

How it works

According to the paper, Flamingo handles arbitrarily interleaved visual and textual data and can ingest images or videos as inputs^[2]. The order and mixture are not fixed in advance: pictures, clips and sentences can appear in whatever arrangement the prompt uses, and the model works from that sequence as a whole.

The model learns new tasks from a few example pairs without additional training^[1]. In practice this means a user supplies a handful of input-output examples inside the prompt, and the model continues the pattern rather than having its weights changed. This is what is meant by few-shot learning: the task is described by example, not by retraining.

The final model has 80 billion parameters and builds on DeepMind's 70-billion-parameter Chinchilla language model^[1]. Rather than training a single network from scratch, it combines separately pre-trained, frozen language and visual models^[1]. "Frozen" here means those pre-trained parts are not updated during this stage; their existing abilities are kept intact and reused. The result is a system whose language side already knows how to produce fluent text, and whose visual side already knows how to represent images, joined into one model that can respond to both at once.

Results

On 16 tasks, with as few as four examples per task, Flamingo beat previous few-shot approaches, and in several cases outperformed fine-tuned, task-specific methods^[1]. The paper makes a related claim in stronger terms: Flamingo outperforms models fine-tuned on thousands of times more task-specific data on numerous benchmarks^[2], and a single Flamingo model achieves a new state of the art with few-shot learning when prompted with task-specific examples^[2].

The comparison matters because it reverses the usual trade-off. The conventional route to strong results on a task is to collect a large labelled dataset for that task and fine-tune a model on it. Flamingo's claims are about the opposite setting: very little task-specific data, supplied as examples in the prompt, with one model covering many tasks.

Where it sits

Flamingo appeared at a moment when large language models were already established and the open question was how to attach other kinds of input to them. Its answer — keep pre-trained language and visual models frozen, and rely on interleaved prompts and in-context examples — made multimodal prompting a practical way to specify tasks rather than a demonstration.

The Flamingo paper was submitted to arXiv on 29 April 2022^[2], a day after the model was announced^[1]. Between them, the announcement and the paper set out both the engineering description — 80 billion parameters on top of Chinchilla, frozen components, interleaved inputs — and the evaluation story of few-shot results across many tasks. For readers following multimodal models, Flamingo is a reference point for the idea that a visual language model can be general-purpose without being retrained for each new task.