Google publishes RT-1

Why general robot control is hard

A conventional robot program is built for a narrow job: an arm that picks one kind of part, a gripper that stacks one kind of block. A new task usually means new engineering, and a controller tuned for one job tends not to carry over to the next. A general-purpose robot would need the opposite: a single model that takes a spoken instruction and produces the right actions across many different objects and tasks.

The authors of RT-1 argued that the key to general robot models lies in two things taken together — open-ended, task-agnostic training, and high-capacity architectures able to absorb diverse robot data^[1]. Neither ingredient is a single clever trick; the claim is about how data and model size are combined.

What was published

On 13 December 2022, Google researchers submitted the paper RT-1: Robotics Transformer for Real-World Control at Scale to arXiv^[1]. The paper presents a model class the authors call the Robotics Transformer, trained on real robot data rather than on hand-written control rules^[1].

The two ingredients

Task-agnostic training means the model is not built around one job. Instead of a separate policy for each task, training covers an open-ended mix of tasks and instructions, so that following an instruction becomes the general skill and individual tasks are instances of it. The model is not specialised in advance to the thing it will be asked to do.

High capacity means a large neural network. The Transformer architecture is built on attention, which lets each part of an input weigh how relevant other parts are; in a robot setting that is what allows words in an instruction to be tied to what is visible in the current camera view, and actions to be produced from that pairing. Capacity sets how much varied data a model can absorb without saturating. The authors' argument is that diverse robot data is only useful if the architecture is large enough to take it in^[1].

The data behind it

To train the model the team collected more than 130,000 episodes over 17 months, using 13 robots^[1]. An episode is one recorded run: what the robot saw, what it was told to do, and what it did. Collecting at that volume is the expensive part of this kind of work — the data has to come from physical machines moving in real rooms, one run at a time. Using 13 robots rather than a single machine means the training set is not tied to one physical setup^[1].

Results and release

The authors reported more than 700 instructions performed at a 97% success rate^[1]. They also released the code for RT-1 alongside the paper^[1].

Why it matters

The paper is a statement about method as much as about a model. Its argument is that general robot control comes from open-ended, task-agnostic training combined with high-capacity architectures, rather than from a controller engineered task by task^[1]. That framing treats robot learning as a data-and-scale problem, in the same family as other large models: gather enough varied experience, and make the network big enough to use it.

Releasing the code with the paper matters for the same reason. A result that depends on a large private robot dataset is hard for others to check; publishing the implementation at least lets other groups see how the model is built and run it on their own data^[1]. The reported 97% success rate over more than 700 instructions is the number the paper puts on the approach^[1], and the two ingredients named in it — task-agnostic data and capacity — are the parts later work in the same direction has to argue with.