Adam: A Method for Stochastic Optimization

What problem it sets out to solve

Training a machine-learning model is usually an optimisation problem: adjust the model's parameters until a loss function becomes small. The standard tool is gradient descent — compute the gradient of the loss with respect to the parameters, which points in the direction the loss grows fastest, and move the parameters the other way. The paper "Adam: A Method for Stochastic Optimization" was first submitted to arXiv on 22 December 2014 as 1412.6980, and was written by Diederik P. Kingma and Jimmy Ba.^[1] It presents Adam as an algorithm for first-order gradient-based optimisation of stochastic objective functions, based on adaptive estimates of lower-order moments.^[1]

Those words describe real constraints. "First-order" means the method uses only the gradient, not second derivatives such as curvature information, which are expensive or impractical to compute for large models. "Stochastic" means the objective is estimated from random samples — in practice small batches of training data — so each gradient is a noisy guess rather than an exact value. "Adaptive estimates of lower-order moments" means the algorithm keeps running estimates of the average gradient (the first moment) and of the average squared gradient (the second moment), and uses them to decide how large a step to take for each individual parameter.

How the method works

The intuition is to divide the signal by its own typical size. For every parameter the algorithm maintains a decaying average of recent gradients, which smooths out the noise and indicates a consistent direction, together with a decaying average of recent squared gradients, which measures how big the gradients for that parameter usually are. The step for a parameter is then roughly the smoothed gradient divided by the square root of the smoothed squared gradient. A parameter that consistently receives large gradients has its step damped; a parameter whose gradients are small or rarely non-zero gets a relatively larger step. The result is that a single global step-size setting is adapted per parameter rather than applied uniformly.

The authors point out that the update rule is invariant to diagonal rescaling of the gradients: if the gradients of one coordinate are all multiplied by a constant, the update for that coordinate is unchanged, because the constant appears in both the numerator and the denominator. The paper highlights this property for problems that are large in data and/or parameters, where different coordinates can naturally have very different scales.^[1] The same construction is what the authors argue makes the method suitable for non-stationary objectives and for problems with very noisy or sparse gradients, where many gradients are zero or nearly so.^[1]

On practical grounds, the authors describe the method as straightforward to implement, computationally efficient, and requiring little memory — it stores a few running averages per parameter, not a matrix of pairwise interactions.^[1] They also describe the hyper-parameters as having intuitive interpretations and as typically requiring little tuning.^[1]

Results

Beyond the algorithm itself, the paper analyses theoretical convergence properties and gives a regret bound on the convergence rate that is comparable to the best known results under online convex optimisation.^[1] A regret bound compares the cumulative loss of the algorithm with that of the best fixed choice made in hindsight; it is a statement about worst-case behaviour over a sequence of objectives, not a guarantee that training a particular model will end at a good solution.

Why it matters

The arXiv record lists the paper as published at the 3rd International Conference on Learning Representations (ICLR) in San Diego in 2015.^[1] What the paper offers is not a new model architecture but a general-purpose tool used every time a model is trained. Its stated design goals — low memory, cheap per-step computation, robustness to noisy and sparse gradients, and hyper-parameters that need little tuning — are exactly the properties that decide whether an optimiser can be used routinely on large problems. The combination of adaptivity per parameter with a light memory footprint is the reason the method is described in the paper as widely applicable, and it is on those grounds that the work is usually discussed.