Google DeepMind introduces RT-2 vision-language-action model

Background

Most robot control systems learn from robot data: demonstrations of a machine carrying out tasks, recorded in a particular lab or with a particular arm. That kind of data is expensive to collect and narrow in coverage, so a policy trained on it tends to work well in the settings it has seen and poorly in new ones. Language and vision models are trained differently, on large collections of text and images taken from the web. The question behind RT-2 is whether that kind of knowledge can be made useful for moving a robot.

What was announced

On 28 July 2023, Google DeepMind introduced Robotic Transformer 2 (RT-2), which the lab described as a first-of-its-kind vision-language-action model ^[1]. The name describes the three things the model handles: what it sees, what it is told in words, and what it does. Its central claim is that it transfers knowledge from web data to robot control ^[1] — that training on web material leaves the model with visual and linguistic competence it can then apply when driving a physical machine.

How it works

The distinctive design choice is how actions are represented. RT-2 expresses actions as tokens in its output, in the same way a language model emits tokens for words ^[1]. A token is the small unit a model predicts one step at a time: in a text model, a fragment of a word; here, a fragment of a motor command. A physical action — how far to move, where to point — is written as a sequence of numbers, and those numbers are treated as just another kind of text the model can generate. The practical consequence is that the same training recipe and the same kind of output layer used for language can be reused to produce behaviour, rather than requiring a separate action head designed by hand.

Evaluation

The evaluation spanned more than 6,000 trials ^[1]. The headline number concerns generalisation to unseen scenarios, which rose to 62% for RT-2 from 32% for RT-1 ^[1]. In other words, when the model met situations that were not part of its robot training, it succeeded at close to twice the rate of the earlier system.

The skills being tested

Google DeepMind defined three skill categories for RT-2: symbol understanding, reasoning and human recognition ^[1]. These separate different demands that language and vision bring to control — reading symbols and text in a scene, working out what a situation calls for, and recognising what a person is doing or asking for. An illustration of the reasoning category in action: the model could decide which object, such as a rock, could be used as an improvised hammer ^[1]. Choosing a substitute tool is not a memorised motion; it requires connecting an instruction and a goal to whatever the scene happens to offer.

Why it matters

The result was presented as evidence for a route around a long-standing constraint in robotics. If competence acquired from web-scale text and images carries over into control, then robot behaviour need not depend entirely on the volume of robot demonstrations a lab can collect. It also points to a way of giving machines instructions in ordinary language rather than through task-specific code, since the model is built to take words and images in and produce actions out. RT-2 was described as first-of-its-kind at the time of its announcement ^[1], which places it at the point where vision-language models and robot control were joined into a single system.