Magnetic control of tokamak plasmas through deep reinforcement learning

The problem

A tokamak holds a hot, ionised gas — a plasma — inside a vessel using magnetic fields, so that it does not touch the walls. Keeping such a plasma in a chosen shape, and keeping it there, is a control problem rather than a purely magnetic one: the currents in the magnetic actuator coils around the vessel must be adjusted continuously, in closed loop, as the plasma moves and changes. That means high-dimensional, high-frequency control, with many coils to command and very little time to decide ^[1]. The authors describe tokamak feedback control as one of the hardest real-world systems reinforcement learning has been applied to ^[1].

Conventionally, such controllers are designed by people who start from physics models and hand-tune how the coils should respond. The paper asks whether a controller can instead learn the policy for commanding the coils by itself.

How it works

The authors present a controller architecture that learns autonomously to command the full set of control coils ^[1]. Reinforcement learning, in general terms, is trial and error with a score: a controller takes actions in an environment, receives a numerical reward describing how well it did, and adjusts its behaviour to get more reward. Here the actions are commands to the coil set, and what the controller is judged on is the plasma state it produces.

A distinctive part of the design is where the human input sits. Objectives are specified at a high level, while physical and operational constraints are still satisfied ^[1]. In other words, the operator states what plasma configuration is wanted rather than writing out current waveforms coil by coil; the learned controller works out the coil commands, and the constraints that keep the machine within safe and workable limits remain in force. This separation matters because it is the part of the problem that is hard to specify by hand — the mapping from a desired plasma shape to a full set of coil currents — that the learning has to fill in.

Results

The experiments were run on the Tokamak a Configuration Variable. The learned controller produced and controlled a range of plasma configurations there, including elongated conventional shapes ^[1]. It also handled advanced configurations: negative triangularity, and the so-called snowflake shapes ^[1]. These are plasma cross-sections that differ from the standard elongated one, and each places different demands on the coils.

Beyond single plasmas, the authors also sustained droplets in which two separate plasmas were maintained simultaneously inside the vessel ^[1]. Two distinct plasma objects in one vessel, each held in place at the same time, is a demanding test of a controller, because the magnetic configuration has to serve both at once.

Why it matters

The paper appeared in Nature on 16 February 2022, volume 602, pages 414 to 419, with Jonas Degrave, Federico Felici, Jonas Buchli and Michael Neunert among the authors ^[1]. Its importance lies in what it demonstrates about learned control in a real physical system rather than a simulation: that a controller trained with reinforcement learning can take on high-dimensional, high-frequency actuation — the whole coil set — inside a tokamak, with objectives set at a high level and physical constraints respected, and can then deliver a variety of plasma shapes, including non-standard ones and the two-plasma case ^[1].

Tokamak operation depends on being able to shape and hold plasmas reliably, so a controller that reaches a wide range of configurations from high-level objectives is a meaningful data point for how such machine control might be approached. The authors' own framing — that this is among the hardest real-world systems reinforcement learning has been applied to — sets the result in the broader story of taking learning-based control out of games and simulations and into instruments where a mistake has physical consequences ^[1].

Limits and open questions

The paper reports controlled experiments on one device and a set of configurations; it does not claim to have solved tokamak operation as a whole. The value of the work is bounded by the setting it was tested in, and the questions it leaves are the ordinary ones for learned controllers in physical systems: how the approach carries to other machines, and how the constraint handling behaves outside the conditions covered in the experiments.