Google announces Gemini 1.5 Pro with a 1-million-token context window

Background

On 15 February 2024, Google announced Gemini 1.5, a new generation of its Gemini model family, starting with Gemini 1.5 Pro.^[1] The timing was tied to a renaming: a week earlier, Bard, Google's chatbot, had become Gemini.^[1] Demis Hassabis announced the new model, and Sundar Pichai wrote an opening note for the announcement.^[1]

What was announced

Gemini 1.5 Pro is a large language model, the kind of system that takes in text (and, in this case, other media) and produces a continuation. Its distinguishing architectural choice was a mixture-of-experts design.^[1] In a mixture-of-experts model, the network is split into many separate sub-networks, usually called experts, plus a routing component that decides, for each piece of input, which few experts should handle it. Only those experts do work on that input. The practical consequence is that a model can be made much larger in total parameter count without its computational cost growing in step, because most of the network stays idle for any given token.

The context window

A context window is the amount of material a model can hold in view at one time — everything it is given plus everything it has generated so far. It is measured in tokens, the chunks of text (roughly word-pieces) that models read and write. Gemini 1.5 Pro offered a 128,000-token window by default, and up to 1 million tokens in a private preview.^[1] A million tokens is far more than a single long report: it is enough for hours of transcript or a small shelf of documents, kept available to the model in one pass rather than retrieved piece by piece.

To show what that meant in practice, Google demonstrated the model analysing the 402-page transcript of the Apollo 11 mission and a 44-minute silent Buster Keaton film.^[1] The two examples were chosen to make different points. The transcript is a long, dense text that no ordinary context window could contain, so questions about it test whether the model can locate and connect details spread across hundreds of pages. The film is not text at all; handling 44 minutes of it silently means the model was taking in a long sequence of images, not just words.

Performance

Google said that 1.5 Pro performed at a broadly similar level to 1.0 Ultra, the largest model of the previous generation.^[1] That comparison is the company's own description rather than an independent measurement, but it framed the release: the new model was presented as matching the top of the old line while being built on a different, mixture-of-experts foundation.

Beyond a million tokens

Google also said it had tested the model with up to 10 million tokens in research.^[1] That figure was not offered as a product capability, and no timeline was attached to it; it served to indicate how far the team believed the approach could be pushed, and that the limit being advertised was a current setting rather than a hard ceiling of the method.

Why it mattered

The announcement moved attention from benchmark scores toward capacity: how much material a model can take in at once, and what kinds of tasks become possible when the answer no longer depends on a retrieval step that chops documents into fragments. Long transcripts, whole codebases, long recordings and multi-document questions are all cases where the model must keep many details in view simultaneously. Making the window itself large, rather than engineering around a small one, changes what can be asked. The mixture-of-experts design matters for the same reason on the cost side: it is the mechanism by which a model of this size could plausibly be served at all. Gemini 1.5 Pro was, in that sense, an argument about direction as much as a single product — an early claim that context length would be a main axis along which frontier models are compared.

^[1]: Google, "Our next-generation model: Gemini 1.5".