Imitation learning

What it is

Imitation learning trains an agent to perform a task by supervised learning from expert demonstrations. It is also called learning from demonstration or apprenticeship learning ^[1]. The premise differs from learning by trial and error: instead of discovering a good way to act, the agent is shown one and asked to reproduce it.

How it works

The most basic form is behaviour cloning, which trains a policy by supervised learning on demonstration data ^[1]. A policy is the rule that maps what the agent senses to the action it takes, and supervised learning means fitting such a rule to labelled examples. In behaviour cloning the labels come from the expert: each recorded moment supplies an input — a camera image, a robot's joint angles, a game state — together with the action the expert took at that moment. Training then uses the same machinery as ordinary classification or regression.

Driving makes the idea concrete. A camera records the road ahead while a human drives, and each frame is paired with the steering angle the human applied. Given enough such pairs, a network can produce plausible steering for frames it has never seen. That is essentially the setting of the earliest named example ^[1].

Where it came from

The earliest named example is ALVINN, in which Dean A. Pomerleau used a neural network trained on human driving demonstrations in 1988 ^[1]. The recipe is the one described above: record what a person does behind the wheel, treat those actions as labels, and let the network learn the mapping from view to action.

Behaviour cloning remained the reference point for later work, which kept the demonstration data but changed how it is used. DAgger, published by Ross, Gordon and Bagnell in 2011, iteratively aggregates expert data to improve behaviour cloning ^[1]. GAIL, published by Ho and Ermon in 2016, matches behaviour distributions using a generative adversarial approach ^[1]: rather than comparing single actions, the adversarial setup compares the overall behaviour produced by the learned policy with the behaviour of the expert. Decision Transformer, published by Chen and colleagues in 2021, recasts reinforcement learning as a sequence modelling problem ^[1], treating a trajectory as a sequence to be predicted rather than as a stream of independent decisions.

Where it is used today

Imitation learning is a common way to build robot controllers, because demonstrations are often easier to obtain than a well-specified reward signal. The RT-1 authors described their method as an imitation learning approach ^[2], placing a large-scale robotics system in this lineage rather than in the reinforcement learning one.

Limitations

The main limitation of behaviour cloning is distribution shift: once the policy deviates from the expert trajectory it enters states that are absent from the demonstration data ^[1]. Human drivers correct small errors constantly; a cloned policy that drifts slightly off-centre finds itself looking at views it was never trained on, and its mistakes compound. Methods such as DAgger attack this directly by adding expert data over successive rounds instead of training once on a fixed set ^[1].

A second limit is the ceiling set by the demonstrations themselves. The RT-1 authors noted that because their method is imitation learning, it may not be able to surpass the performance of its demonstrators ^[2]. A cloned policy can be made reliable and cheap to train, but it inherits the skill level of whoever supplied the data.