Deep Sparse Rectifier Neural Networks

The question the paper takes up

Training a deep neural network that is purely supervised — no unsupervised pre-training step in front of the supervised objective — was a recognised difficulty, and the paper presents its results as a milestone in understanding that difficulty ^[1]. The authors ask what happens when the standard squashing nonlinearities are replaced by rectifying neurons, and whether such a network can be trained well when the labels are plentiful ^[1].

What a rectifying neuron is

A rectifying neuron passes part of its input through unchanged and clamps the rest to zero: for negative inputs the output is zero, for positive inputs the output rises linearly. This is a hard nonlinearity, and at exactly zero it has no derivative in the usual sense. Both properties distinguish it from the logistic sigmoid and the hyperbolic tangent, which bend smoothly and never sit at a flat zero over a whole range of inputs. Because a rectifying neuron outputs an exact zero for every negative input, a layer of such neurons produces a representation in which many units are genuinely off rather than merely small.

What the authors report

The abstract makes four claims. First, rectifying neurons are an even better model of biological neurons than logistic sigmoid or hyperbolic tangent neurons ^[1]. Second, networks built from them yield equal or better performance than hyperbolic tangent networks, and they do so despite the hard non-linearity and the non-differentiability at zero ^[1]. Third, rectifying neurons create sparse representations with true zeros, which suit naturally sparse data ^[1]. Fourth, deep rectifier networks can reach their best performance without unsupervised pre-training on purely supervised tasks that come with large labelled datasets ^[1]. Taken together, these results are framed as a milestone in understanding the difficulty of training deep but purely supervised neural networks ^[1].

Where it was published

The paper, by Xavier Glorot, Antoine Bordes and Yoshua Bengio, appears in the Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, held in 2011 ^[1]. It is printed on pages 315 to 323 of volume 15 of Proceedings of Machine Learning Research ^[1]. The conference took place in Fort Lauderdale, Florida, on 11 to 13 April 2011, and the proceedings were edited by Geoffrey Gordon, David Dunson and Miroslav Dudik ^[1].

Why it matters

Two of the paper's points carry particular weight. The comparison with biological neurons puts the choice of activation function on a different footing from mere computational convenience: the rectifier is presented as the more faithful model of what a neuron does ^[1]. The claim about unsupervised pre-training matters for practice, because it says that when labelled data are abundant the extra unsupervised stage is not needed for a deep rectifier network to reach its best performance ^[1]. And the sparsity claim explains why exact zeros in a representation can be an asset rather than a defect when the data themselves are sparse ^[1]. The paper's own framing is modest and diagnostic: it treats these findings as a step toward understanding why deep supervised networks are hard to train, not as the end of that inquiry ^[1].