Proximal Policy Optimization Algorithms

The problem

Reinforcement learning is the branch of machine learning in which an agent learns by interacting with an environment and receiving rewards. It is not given the correct answer for every input; it tries actions, observes the consequences, and adjusts its behaviour. The policy is the rule that maps situations to actions, and policy optimization methods try to improve that rule directly so that actions associated with higher reward become more likely.

On 20 July 2017, John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov submitted "Proximal Policy Optimization Algorithms" to arXiv^[1]. The paper introduces proximal policy optimization, or PPO. In the abstract, the authors present the proposed methods as much simpler to implement, more general, and empirically better in sample complexity^[1]. Sample complexity refers to how much interaction data an algorithm needs to reach a given level of performance; when data collection is expensive, a lower requirement matters.

How it was tested

The paper evaluates PPO on benchmark tasks including simulated robotic locomotion and Atari game playing^[1]. Such benchmarks are used to compare learning algorithms on control and game-playing problems. The abstract reports that PPO outperforms other online policy gradient methods^[1]. Online policy gradient methods learn from data collected by the current policy while it is being improved, rather than from a fixed dataset; policy gradients adjust the policy's parameters in a direction that makes higher-reward actions more likely.

The paper's claims are comparative and empirical. It does not rest on a single demonstration but on a set of benchmark tasks, and its case for the method is framed around practical properties: implementation simplicity, generality, and sample efficiency^[1].

Results

The abstract's summary of results is that the proposed methods are simpler, more general, and better in sample complexity, and that PPO outperforms other online policy gradient methods^[1]. OpenAI's later description makes a similar practical point: PPO performs comparably to or better than state-of-the-art approaches while being much simpler to implement and tune^[2].

Why it matters

According to OpenAI, PPO became the default reinforcement learning algorithm at the organisation because of its ease of use and good performance^[2]. That is a notable form of adoption: a method becomes the standard choice for repeated experimental work, not just a result reported once in a paper.

The OpenAI Baselines release includes PPO2, a GPU version that runs approximately 3x faster than the PPO baseline on Atari^[2]. PPO2 shows the algorithm being packaged for practical experimentation: the same method, reimplemented to run faster on hardware used for training.

Later influence

OpenAI's account places PPO among its standard tools: it became the default reinforcement learning algorithm there, and the Baselines release supplied PPO2^[2]. The original paper sets out the method's case in terms of simpler implementation, greater generality, better sample complexity, and stronger performance than other online policy gradient methods^[1]. Those are the properties OpenAI highlights when explaining why the method was adopted^[2].