Dropout: A Simple Way to Prevent Neural Networks from Overfitting

The problem the paper takes on

Deep neural networks with a large number of parameters are powerful, but they suffer seriously from overfitting ^[1]. Overfitting means the network fits the training examples closely while doing worse on data it has not seen: it has picked up peculiarities of the training set instead of regularities that carry over.

The classic remedy is an ensemble. Train many networks separately and average their predictions, and the errors of individual networks tend to cancel. That remedy runs into a practical wall: large networks are slow to use, which makes it hard to control overfitting by combining predictions of many large nets at test time ^[1]. The cost of an ensemble grows with the number of members, both in training and in the work needed for each prediction.

"Dropout: A Simple Way to Prevent Neural Networks from Overfitting" appeared in volume 15, issue 56, of the Journal of Machine Learning Research, pages 1929 to 1958, in 2014. Its authors are Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov ^[1].

Randomly dropping units

The key idea is to randomly drop units, along with their connections, from the network during training ^[1]. At each training step every unit is kept or removed according to chance; a removed unit passes nothing on and receives no weight update for that step.

The reason has to do with co-adaptation. In an ordinary network a unit can come to rely on particular other units being present and correct, so a brittle arrangement of features can fit the training data. Dropping units prevents them from co-adapting too much ^[1]: no unit can count on any given partner, so each is pushed toward features that are useful in many different company.

Each random pattern of dropped units describes a different thinned network, one that shares weights with its siblings. During training, dropout therefore samples from an exponential number of different thinned networks ^[1]. With n units there are 2^n ways to keep or drop them, so a single training run touches an enormous family of subnetworks rather than one.

Assembling the ensemble at test time

Using all those thinned networks at prediction time is impossible. The paper's answer is compact: at test time a single unthinned network with smaller weights approximates the effect of averaging the predictions of all those thinned networks ^[1]. The full network is used, with every unit present, but its weights are scaled down to compensate for the fact that more units are active than during training. One model, one forward pass, and behaviour close to the average of the sampled subnetworks.

Results

The authors report state-of-the-art results on many benchmark datasets in vision, speech recognition, document classification and computational biology ^[1]. That range matters: dropout is not tied to one architecture or one kind of data. It is a change to training, not to the shape of the network or to what the network can represent at test time.

Why it matters

The appeal is the ratio between what is added and what is gained. Nothing about the model has to be redesigned; a training loop gains a random mask and a scaling factor. And an ensemble effect, usually paid for with many full trainings, is obtained from a single one — which is precisely the obstacle the paper set out to remove for large networks ^[1].