Long context

What a context window is

When a language model is asked to do something, it does not work on the last sentence alone: it works on everything included in that request — instructions, documents, images, transcripts. This material is the context, and the context window is the ceiling on how much of it the model can take in at once, measured in tokens, the units (roughly word fragments) into which text is split.

A long context is simply a very large ceiling. Where a small window forces material to be chopped into chunks, summarized and stitched back together, a long-context model is designed to accept whole books, long codebases or hours of recorded audio and video in one go. A user can hand over the entire thing and ask a question about one line inside it.

The ability being tested is therefore not only storage but use: can the model locate a detail buried in a long input and reason with it?

Position matters as much as length

A bigger window does not automatically mean the model uses it well. Liu et al. showed in 2023 that accuracy drops sharply when the information needed to answer sits in the middle of a long input rather than at the beginning or the end ^[2]. The finding held even for models marketed specifically as long-context models, which is why long-context evaluation now tests position sensitivity and not only maximum length ^[2]. A model can pass the question "how many tokens can you accept" and still fail the question "can you find the sentence on page 300".

The scale reached in 2024

Google's Gemini 1.5 technical report, published in March 2024, described a family of multimodal models able to recall and reason over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio ^[1]. The family launched with two models: an updated Gemini 1.5 Pro and the lighter Gemini 1.5 Flash ^[1].

Google reported near-perfect recall on long-context retrieval tasks across modalities, and said next-token prediction and retrieval above 99% continued to improve up to at least 10 million tokens ^[1]. For comparison, the same report put Claude 3.0 at 200k tokens and GPT-4 Turbo at 128k tokens ^[1], well below that figure. Google also said Gemini 1.5 improved the state of the art in long-document question answering, long-video question answering and long-context automatic speech recognition ^[1].

Two demonstrations in the report show what such a window is for. Given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learned to translate English into Kalamang at a level similar to a person who had learned from the same content ^[1]. And in professional use cases across 10 job categories, Gemini 1.5 working together with users produced reported time savings of 26% to 75% ^[1].

Where it is used

Long context changes what can be asked of a model in one step: reading and querying a whole contract, searching across a set of documents at once, answering questions about a recorded lecture or meeting, transcribing speech with the help of a long glossary, or picking up a task from a manual instead of from many labelled examples. The Kalamang result is an instance of the last pattern — the "examples" are the reference material itself.

Limits

Length is a capability, not a guarantee. The drop in the middle of the input found by Liu et al. is still the reason evaluations check where in the input an answer sits ^[2]. Feeding more material in also tends to cost more computation and time, so filling every request to the maximum is usually wasteful; retrieving the relevant few pages first still does work that a large window does not replace.