Reinforcement learning

What it is

Reinforcement learning is the branch of machine learning and optimal control in which an agent learns to take actions in a dynamic environment in order to maximise a reward signal^[1]. The setting differs from the more familiar supervised case: the learner is not handed a data set in which every example carries the right answer. Instead the agent acts, the environment responds, and a reward — sometimes delayed, sometimes sparse — says how well things went. The field frames its central problem as the exploration–exploitation dilemma, and usually models the environment as a Markov decision process^[1].

The vocabulary is worth fixing early. The agent is the learner and decision maker; the environment is everything it interacts with; the policy is the rule the agent follows to pick actions; the reward is the number it is trying to accumulate.

How it works

The loop is simple to state. At each step the agent observes something about the state of the environment, chooses an action, and receives a reward together with a new state. A Markov decision process packages this as a set of states, a set of actions, the probabilities of moving between states under each action, and the rewards attached to those transitions. The "Markov" part means the current state is treated as a sufficient summary of the past for the purpose of deciding what to do next.

Two problems dominate the design of any algorithm. The first is the exploration–exploitation dilemma: the agent can either try actions whose value it does not yet know, hoping to discover something better, or stay with the actions that have paid off so far^[1]. Spend all the time exploring and reward is wasted; never explore and the agent may settle into a mediocre habit. The second is credit assignment: when a reward arrives long after the actions that caused it, the algorithm has to work out which of those actions deserves the credit.

A small example makes this concrete. Imagine a program steering a token through a simple grid towards a goal square. It is rewarded on reaching the goal and not otherwise. Early on it moves at random; a few runs end at the goal, and the actions taken on those runs become slightly more likely. Over many runs the policy shifts towards the short path. Nothing in that description requires a labelled example of the correct move at any square, and nothing tells the agent in advance which of its earlier steps mattered — it has to work that out from the rewards it eventually receives.

Where it came from

The field grew out of work that treated learning as a problem of temporal credit assignment. Sutton's 1984 doctoral thesis was titled Temporal Credit Assignment in Reinforcement Learning^[1]. In 1987 Williams proposed the REINFORCE gradient estimator, a way of estimating how the parameters of a policy should be adjusted so that behaviour that earned reward becomes more probable^[1]. Watkins proposed Q-learning in his 1989 doctoral thesis, Learning from Delayed Rewards^[1]; Q-learning estimates the value of taking a given action in a given state, and learns that estimate from experience of the rewards that followed.

In use today

Reinforcement learning now reaches beyond simulated environments into machines that act physically. On 17 November 2025 Physical Intelligence published π*0.6, describing a method for training generalist robot policies with reinforcement learning to improve success rate and throughput on real-world tasks^[2]. In China, the China Securities Journal reported that XPENG's IRON humanoid used an end-to-end large model together with reinforcement learning algorithms for walking^[3]. In both cases the appeal is the same: a system that can improve from its own attempts on a task, rather than only from examples prepared in advance.

Limits

Wikipedia lists low sample efficiency, instability and convergence problems, poor generalisation and reward-design issues among the limitations of reinforcement learning^[1]. Each has a plain reading. Low sample efficiency means the agent may need a very large number of interactions before it behaves well, which is expensive when those interactions happen on real hardware. Instability and convergence problems mean that training does not always settle down, and that outcomes can depend heavily on how the learning is set up. Poor generalisation means performance can drop when the situation differs from what was encountered during training. Reward design is the quietest problem and often the sharpest: a reward is only a stand-in for what we actually want, and an agent will optimise the stand-in, including its imperfections.