RT-2 (Robotic Transformer 2)

What RT-2 is

RT-2 (Robotic Transformer 2) is a vision-language-action model from Google DeepMind, announced on 28 July 2023 ^[1]. The name describes the three things it puts together: it takes in what a robot's camera sees, reads a written instruction, and produces the robot's next action. Google DeepMind calls this combination a vision-language-action (VLA) model ^[1].

Where it came from

RT-2 was built by co-fine-tuning two existing vision-language models, PaLI-X and PaLM-E, on robot trajectory data ^[1]. A vision-language model is one trained on large collections of images paired with text; fine-tuning continues that training on a narrower dataset. Here the extra data were trajectories, the recorded motion of a robot working through a task. The arXiv paper describing the work was submitted on 28 July 2023 and lists Anthony Brohan and 53 other authors ^[2]; in it, the authors describe co-fine-tuning state-of-the-art vision-language models on robotic trajectory data ^[2].

Actions written as text

A distinguishing design choice is that RT-2 outputs robot actions as text tokens ^[1][2]. A token is a small chunk of text, roughly a word or a piece of one, and text models work by predicting one token after another. If an action can be written down as a short string of characters, the same machinery can produce it. Because the model learns from both web data and robotics data ^[1][2], knowledge picked up from images and text on the web can in principle flow into how the robot moves.

Results

Google DeepMind evaluated the model in more than 6,000 robotic trials ^[1]. On scenarios the model had not seen, performance rose to 62%, compared with 32% for its predecessor RT-1 ^[1]. RT-2 also kept its performance on the original tasks present in the robot data ^[1], so the gain on unfamiliar situations did not come at the cost of the tasks it had been trained on.

Three kinds of skill

The team sorted the abilities it tested into three categories: symbol understanding, reasoning, and human recognition ^[1]. Reasoning is the most visible of the three in the demonstrations. RT-2 could carry out instructions that require a step of inference, such as picking up an object that could serve as an improvised hammer ^[1]. The instruction does not name a hammer as such; the model has to connect the idea to whatever object is in front of it.

What it represents

RT-2 sits where two kinds of training data meet: material collected from the web, and recordings of real robot motion ^[1][2]. Its reported result is a statement about transfer, that a model drawing on web data as well as robot data can handle unfamiliar situations better than a model built only around robot data. It is also why the VLA label matters: it names a class of model in which seeing, reading and acting are handled by one network rather than by separate pipelines bolted together.