Batch normalization

What it is

Batch normalization is a technique that re-centres the inputs of a layer to zero and rescales them to unit variance, which makes training faster and more stable ^[1]. Rather than computing statistics over the whole training set, it normalises the activations of each layer per dimension over a mini-batch ^[1].

How it works

Training a deep network means updating the parameters of many layers at once. Whenever the earlier layers change, the distribution of inputs arriving at a later layer shifts with them, so the later layer is effectively chasing a moving target. Keeping the scale of those inputs under control is what batch normalization is for.

A concrete example: suppose a layer produces activations for a small batch of samples, and each sample has one value in a given dimension. Batch normalization collects the values that the whole mini-batch produced in that dimension, computes their mean and variance, then subtracts the mean and divides by the standard deviation. The output in that dimension is pulled back to around zero with a scale of about one ^[1]. The normalisation is per dimension and per mini-batch, and the statistics come from that mini-batch, not from the full training set ^[1]. Nor is it a one-off preprocessing step: the normalisation is built into the model architecture and performed for each training mini-batch ^[2].

Where it came from

Sergey Ioffe and Christian Szegedy introduced the method in 2015, in the paper "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" ^[1]. First submitted in February 2015, the paper frames the difficulty of training deep networks as internal covariate shift: the change in each layer's input distribution as the previous layers' parameters change ^[2]. Its remedy is to make normalisation part of the model architecture and to perform the normalisation for each training mini-batch ^[2].

What it changed

The authors report that this allows much higher learning rates and less careful initialisation, and that it also acts as a regulariser, in some cases eliminating the need for dropout ^[2]. Batch normalization allows higher learning rates without causing vanishing or exploding gradients ^[1], and it makes the network less sensitive to the choice of initial parameters and learning rate ^[1]. Applied to a state-of-the-art image classification model, it achieved the same accuracy with 14 times fewer training steps ^[2].

A revised explanation

The original paper attributed the benefit to reducing internal covariate shift ^[1]. In 2018, Santurkar and colleagues published experiments, including a VGG-16 with added noise, that contradicted that explanation and argued that the real cause is a smoother loss landscape ^[1]. The method kept working; the story told about why it works did not survive intact.

Limitations and relatives

Because it normalises across a batch, batch normalization makes samples within a batch no longer independent and identically distributed, which can degrade gradient estimates ^[1]. A related method, layer normalization, was proposed by Ba, Kiros and Hinton in 2016 ^[1].