Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

What the report set out to do

The report is credited to the Gemini Team at Google and was submitted to arXiv on 8 March 2024 as 2403.05530.^[1] Its subject is a family of multimodal models built around a single claim: that a model can recall and reason over fine-grained information drawn from millions of tokens of context, including several long documents at once, or hours of video and audio.^[1]

Context length is the amount of material a model can hold in view while answering. The report measures the size of that jump against two contemporary models, describing its 10-million-token context as a generational leap over Claude 3.0 at 200K tokens and GPT-4 Turbo at 128K.^[1] A token is the small unit of text, image or audio that a model reads and writes; a few million of them correspond to shelves of documents rather than to a single prompt.

How the models are built

Two models are covered. Gemini 1.5 Pro is an updated version of Pro; Gemini 1.5 Flash is a lighter model designed for efficiency, accepting only minimal regression in quality in exchange.^[1]

Gemini 1.5 Pro is described as a sparse mixture-of-experts Transformer.^[1] A Transformer is the architecture behind most large language models: it lets each part of the input weigh its relation to the other parts. Mixture-of-experts is a way of spending compute selectively. Rather than running every parameter on every token, the model routes each token to a small subset of specialised sub-networks, so the total parameter count can be large while the computation per token stays comparatively modest. The word sparse refers to that selective routing. The report's efficiency argument rests on this design choice.

What the evaluations showed

The central test is needle-in-a-haystack retrieval: a specific piece of information is hidden somewhere inside a very long context, and the model has to find it and report it back. On these tests Gemini 1.5 Pro kept recall above 99.7% up to 1 million tokens across text, video and audio.^[1]

The same recall held when the context was pushed much further: to 10 million tokens of text, about 7 million words; to 9.7 million tokens of audio, up to 107 hours; and to 9.9 million tokens of video, up to 10.5 hours.^[1]

Two smaller studies appear in the report. In one, professionals worked with Gemini 1.5 on their own tasks, and the model delivered time savings of 26% to 75% across 10 job categories.^[1] In the other, the model was given only a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, and learned to translate English into it at a level similar to a person who had learned from the same material.^[1]

Why it matters

Long context changes how such a model is used. When the window is small, long material usually has to be split into chunks, searched for relevant passages and fed to the model piece by piece, and the splitting can cut across the links between paragraphs; a model that keeps its recall over millions of tokens can take the whole document at once.^[1] The same recall over audio and video means several hours of recording can be examined directly instead of being transcribed first.^[1]

The Kalamang result points at something else: with very little material to work from, the model can reach a translation level close to that of a human learner given the same grammar manual.^[1] And the pairing of Pro with the lighter Flash shows the capability being offered at two points on the quality-efficiency trade-off rather than one.^[1]