Understanding the difficulty of training deep feedforward neural networks

The question

Deep feedforward networks stack several hidden layers of units between the input and the output. For a long time such networks could not be trained successfully at all: the authors write that before 2006 deep multi-layer neural networks were not successfully trained, and that since then several algorithms had managed to train them by using new initialization or new training mechanisms ^[1]. Their paper asks a narrower question — what exactly goes wrong at the start of training, when the weights are drawn at random and a standard activation function is used? ^[1]

Why the logistic sigmoid saturates

The paper singles out the logistic sigmoid activation. It argues that this activation is unsuited for deep networks with random initialization, because its mean value can drive especially the top hidden layer into saturation ^[1]. Saturation means that a unit's output barely changes as its input varies: the response curve flattens out, the error signal flowing back to the unit becomes small, and the weights feeding it therefore change slowly.

The authors add a detail that matters for reading training curves: saturated units can move out of saturation by themselves, albeit slowly. This, they suggest, explains the plateaus sometimes seen when training neural networks — stretches in which progress almost stops before it resumes ^[1].

The Jacobian of each layer

The second ingredient is the conditioning of the transformation that each layer applies. The paper states that training may be more difficult when the singular values of the Jacobian associated with each layer are far from 1 ^[1]. A layer's Jacobian collects the derivatives of its outputs with respect to its inputs, and its singular values describe how strongly the layer stretches or shrinks different directions of the input space. If those values sit far from 1, a signal is systematically amplified or damped as it passes through the layer; across many layers the scale of activations, and of the gradients they produce, can drift far from what training needs.

The proposed scheme

Based on these considerations, the authors propose a new initialization scheme that brings substantially faster convergence ^[1]. The scheme follows from the two observations above: the scale of the initial weights is chosen with the saturation behaviour of the sigmoid and with the conditioning of the layer Jacobians in mind ^[1].

Where the paper appeared

The paper appears in the Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR volume 9, pages 249–256, 2010 ^[1]. The conference was held 13 to 15 May at Chia Laguna Resort in Sardinia, Italy, and the proceedings were edited by Yee Whye Teh and Mike Titterington ^[1]. Both authors are listed at DIRO, Université de Montréal ^[2].

Why it matters

What the paper offers is a way of splitting a practical nuisance into analysable parts. Plateaus and stalled runs are not treated as accidents; they are tied to properties that can be named and measured — the saturation behaviour of the activation function and the singular values of the per-layer Jacobians ^[1]. And the analysis is not left as description: it turns into a concrete rule for setting the starting weights, with the reported effect of substantially faster convergence ^[1].