MuZero

What MuZero is

MuZero is a reinforcement-learning algorithm from DeepMind. It was first described in a preprint posted to arXiv on 19 November 2019.^[1] Reinforcement learning is a way of training a system by letting it act, receive rewards and adjust its behaviour. MuZero's distinctive design combines tree-based search with a learned model.^[1] That model is applied iteratively to predict three things: the reward, the action-selection policy and the value function.^[1] Because the model is learned, MuZero does not need to know the environment's underlying dynamics.^[1]

How the learned model is used

Tree-based search explores possible sequences of actions and uses estimates of future reward to choose among them. In MuZero, the search is guided by predictions from the learned model.^[1] The model predicts reward, action-selection policy and value function; applying it repeatedly lets the system evaluate possible futures.^[1] This is the core of the design: planning happens through a model that is itself learned from experience, rather than through a model supplied in advance.^[1]

Results on games

On 57 Atari games, MuZero set a new state of the art.^[1] DeepMind later said that MuZero set a new state of the art on the Atari benchmark.^[3] In Atari tests, the MuZero Reanalyze variant used the learned model 90% of the time.^[3] On Go, chess and shogi, MuZero matched AlphaZero's superhuman performance without being given the game rules.^[1] The board-game result covers three different games, while the Atari result shows the same approach working in visually complex games.^[1]

Publication and team

The arXiv preprint lists 12 authors, including Julian Schrittwieser, Demis Hassabis and David Silver.^[1] A DeepMind post dated 23 December 2020 says that MuZero is described in a paper in the journal Nature.^[3] The post also reports the Atari state-of-the-art result.^[3]

Beyond games

DeepMind later applied MuZero to video compression for sites such as YouTube, reporting an average 6.28% reduction in bitrate.^[2] That application takes the algorithm outside games and into a practical data problem: reducing the number of bits needed to send video.^[2]

Place in the field

MuZero's reported results matter because they cover board games where planning is central and Atari games where the input is visually complex.^[1] In both settings, MuZero learns a model and uses search over that model without being given the rules of the environment.^[1] Its Atari state of the art and its match with AlphaZero in Go, chess and shogi are the concrete results reported for it.^[1] The Reanalyze variant's use of the learned model 90% of the time in Atari tests is one detail of how the system was evaluated.^[3]