Dropout

What dropout is

Dropout is a regularisation technique for artificial neural networks. During training it randomly sets the inputs or outputs of some neurons to zero.^[1] The aim is to stop neurons from developing complicated co-adaptations to the training data, and in that way to reduce overfitting.^[1] In plain terms: a network that has memorised its training set often does so by leaning on very specific combinations of units firing together. If any of those units may disappear at any moment, that kind of reliance stops being dependable, and the network is pushed towards features that are useful on their own.

Dropout belongs with the standard ways of constraining a network so that it generalises rather than memorises. CS231n describes it as a regularisation technique that complements L1, L2 and maxnorm regularisation.^[2]

How it works

At training time, dropout keeps a neuron active with some probability p and sets it to zero otherwise.^[2] The choice is random and made afresh on each pass, so the network cannot count on any particular unit being there.

A small picture helps. Suppose some units in a layer have each learned to detect a slightly different fragment of a pattern, and a unit above them only fires when two of them are present together. Without dropout that pairing is a dependable signal, and the upper unit can afford to be useless on its own. With dropout neither partner is guaranteed, so the upper unit has to learn something that works when either one is missing, and each lower unit has to become useful by itself rather than only in company.

During training, dropout can also be interpreted as sampling a sub-network inside the full network and updating only the parameters of the sampled network.^[2] The weights are shared across all these samples, so a single set of parameters ends up being trained by many different thinned versions of the same network.

The keep probability p is a setting that has to be chosen. CS231n notes that p = 0.5 is a reasonable default but can be tuned on validation data.^[2]

At test time

Dropout is normally switched off at inference time.^[1] At test time no units are dropped.^[2] Because the training procedure and the test procedure then differ in scale, an implementation has to reconcile the two. In the recommended inverted-dropout form, the scaling is done at training time, so the forward pass at test time is left untouched.^[2]

Where it came from

Geoffrey Hinton and colleagues introduced the technique under the name dropout in 2012, in the paper 'Improving neural networks by preventing co-adaptation of feature detectors'.^[1] An earlier related idea, called dilution, appears in the 1991 book Introduction to the Theory of Neural Computation by Hertz, Krogh and Palmer.^[1] CS231n, for its part, credits the technique to Srivastava et al. and the paper 'Dropout: A Simple Way to Prevent Neural Networks from Overfitting'.^[2]

Google holds a patent on the technique, US 9406017B2, although its validity is questionable because of that earlier work.^[1]

Where it sits

Dropout is one tool among several rather than a replacement for the rest. CS231n treats it as a complement to L1, L2 and maxnorm regularisation, and the keep probability is a hyperparameter to be settled against validation data rather than fixed in advance.^[2] The distinction between training and test behaviour is part of the technique as usually implemented: units are dropped while the network is being trained and nothing is dropped once it is being evaluated.^[1][2] And the history matters for how the idea is credited: a name introduced in 2012, a related notion published much earlier, and a patent whose validity is open to question on the strength of that earlier work.^[1]