RT-1 (Robotics Transformer)

What it is

RT-1 (Robotics Transformer) is a model for controlling real robots that turn camera images and a language instruction into actions. The paper, titled "RT-1: Robotics Transformer for Real-World Control at Scale," was submitted to arXiv on 13 December 2022 as arXiv:2212.06817^[1].

The authors describe RT-1 as an imitation learning method: the model learns from demonstrations rather than from a hand-designed reward signal, so what it can do is bounded by what its demonstrators did. The authors say so directly, noting that an imitation learning method may not be able to surpass the performance of its demonstrators^[1].

How it was trained

The capability of RT-1 rests on the scale of its data. The authors trained it on a dataset of more than 130,000 episodes collected over 17 months with 13 robots^[1]. Data gathered this way covers many tasks, objects and arrangements rather than demonstrations recorded for a single task.

The authors also reported that RT-1 could absorb heterogeneous data — from simulation and from other robot morphologies — without sacrificing performance on its original tasks^[1]. Data from different sources can therefore be pooled: adding it does not wash out what the model already knew.

Results

The headline number in the paper is that RT-1 performed more than 700 instructions at a 97% success rate^[1]. The count refers to distinct instructions the model was asked to carry out, not to repeated trials of one task.

The authors also tested longer chains of behaviour. In the SayCan framework, RT-1 could execute long-horizon tasks with as many as 50 steps^[1]. A long-horizon task is one that needs many actions in sequence — find an object, pick it up, carry it somewhere, put it down — and the more steps there are, the more an early mistake can derail everything after it.

Limits

The authors were candid about where generalization stops. RT-1's generalization to new instructions was limited to combinations of concepts it had already seen, and it could not yet generalize to a completely new motion^[1]. If a kind of movement never appeared in the training data, the model will not produce it just because the request sounds reasonable.

The paper also contains an experiment on data diversity. Removing 25% of the tasks while keeping 97% of the data hurt generalization as much as cutting the dataset size by up to 49%^[1]. Coverage of task types matters as much as raw volume: the same number of episodes drawn from fewer kinds of tasks teaches the model less.

Release and later work

The authors released the code for RT-1 to the research community^[1]. Work built on it followed: the authors of OpenVLA listed RT-1-X, a model trained on the Open X-Embodiment dataset, among the prior generalist policies that OpenVLA outperformed^[2]. RT-1 thus became one of the reference points against which later general-purpose robot policies were measured.