Gato

What Gato is

Gato is a generalist agent from DeepMind, announced on 12 May 2022.^[1] It is built around a single network rather than a collection of separate specialist systems. The paper accompanying the announcement, submitted to arXiv on 12 May 2022, describes Gato as a multi-task, multi-embodiment generalist policy.^[2] In plain terms, multi-task means the model is trained and evaluated on many different tasks, and multi-embodiment means the same model is meant to act through different physical or simulated bodies. The central claim is that one set of weights can serve across these settings.^[1]

How it works

During training, data from different tasks and modalities is serialised into a flat sequence of tokens.^[1] This is the key mechanical idea: instead of keeping separate pipelines for text, images, games and robot control, Gato turns the training data into one long stream of tokens. A token is a small unit of information the network reads or writes. By mixing such units in a single sequence, the model can learn from heterogeneous data with one training setup.^[1]

At action time, the output format is also handled through tokens. The paper says Gato decides from its context whether to output text, joint torques, button presses or other tokens.^[2] Context here means the input the model is given and the task it is facing. The model is not limited to producing words: depending on the setting, its next tokens can stand for language, for robot control commands, or for game actions. This is how the same network can be applied across modalities and embodiments.^[1]

What it can do

DeepMind gives several examples of tasks for Gato: playing Atari games, captioning images, chatting, and stacking blocks with a real robot arm.^[1] These examples span vision, language, game playing and robotic manipulation. They are deliberately different from one another, which makes the generalist claim concrete: the same network with the same weights is used across these different tasks and embodiments.^[1]

Evaluation

DeepMind charted how many tasks Gato performed above various percentages of expert score, grouped by domain.^[1] This kind of chart reports a distribution of results rather than a single headline number. It shows, domain by domain, how many tasks crossed thresholds defined relative to expert performance. In the paper, the system is framed as a generalist policy rather than a narrow model for one benchmark.^[2]

Why it matters

Gato's importance lies in the combination it attempts. The same network with the same weights handles tasks across different modalities and embodiments.^[1] That means language, image captioning, Atari play and real robot-arm stacking are treated as parts of one training and action problem, not as unrelated applications. The paper's description of Gato as a multi-task, multi-embodiment generalist policy summarises this aim.^[2]

The model also shows why token-based interfaces matter for generalist agents. If observations and actions from many domains can be represented as tokens, then a single sequence model can, in principle, decide what kind of output to produce from context.^[2] Gato is a generalist agent announced in 2022, with one network and one set of weights applied across tasks, modalities and embodiments.^[1]