Neural Ordinary Differential Equations

A stack of layers, or a curve?

Most deep networks are defined as a discrete sequence of layers: the input passes through the first layer, then the second, and so on, and the number of layers is fixed once the architecture is chosen. "Neural Ordinary Differential Equations", first submitted to arXiv on 19 June 2018 by Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud, asks what happens when that sequence is replaced by something continuous^[1].

What the paper proposes

The key move is a change in what the network is asked to output. Rather than specifying a discrete sequence of hidden layers, the authors parameterise the derivative of the hidden state with a neural network^[1]. An ordinary differential equation describes how a quantity changes from moment to moment; given a starting point, solving the equation tells you where the quantity ends up. Here the neural network plays the role of that rule of change: at any point in the hidden state it says how the state should move, and the output of the model is computed by a black-box differential equation solver that integrates the derivative from the input to the result^[1]. Depth, in other words, becomes a continuous variable — the analogue of "how far to integrate" rather than "how many layers to stack".

The paper highlights three properties that follow from this formulation: the models have constant memory cost, they adapt their evaluation strategy to each input, and they can trade numerical precision for speed^[1]. The memory point is about what has to be kept on hand while the model runs: a stack of discrete layers stores every intermediate activation, while a solver can move through the integration without that growing ledger. The adaptivity point is about step size — a solver can take small steps where the dynamics are awkward and large steps where they are smooth, so different inputs get different amounts of computation. The precision point is the dial on the solver: looser tolerance means faster, rougher answers; tighter tolerance means slower, more accurate ones.

What the authors demonstrated

The authors show these properties on continuous-depth residual networks and on continuous-time latent variable models^[1]. They also construct continuous normalising flows, a generative model that trains by maximum likelihood without partitioning or ordering the data dimensions^[1]. That last part matters because normalising flows need to track how a model warps probability density, and the usual way of keeping that computation tractable is to impose structure on how the dimensions of the data are split up and handled in sequence. The continuous formulation written down in this paper avoids that requirement.

Training through a solver

A model built on a differential equation is only useful if it can be trained, and the obvious obstacle is that the solver sits between the parameters and the loss. The authors show how to backpropagate scalably through any ODE solver without access to its internal operations^[1]. The solver can therefore be treated as a black box: training does not require recording what the solver did step by step, and it does not tie the method to one particular solver.

Why it matters

The paper's contribution is conceptual as much as practical. It redraws the boundary between deep learning and the numerical study of differential equations, so that ideas about integration, step size and tolerance become part of how a network is specified and run. It gives a concrete answer to the question of what a "very deep" model would mean if depth were not counted in whole layers, and it delivers that answer together with a way to train it. The three properties it advertises — constant memory, input-dependent computation, and a precision-speed dial — point at a different set of trade-offs than the ones a fixed stack of layers offers. And the continuous normalising flows sketched in the paper open a route to generative models whose training objective is plain maximum likelihood without the architectural contortions that usually accompany it^[1]. Later work picked up the framing of continuous depth and the use of solvers inside the model, making the paper a reference point for models whose computation is defined by a differential equation rather than by a list of layers.