Xavier (Glorot) initialization

What it is

Xavier initialization, also called Glorot initialization after its first author, is a way of choosing the starting values of a neural network's weights so that training converges substantially faster than with plain random initialization ^[1]. It comes from a paper by Glorot and Bengio, which PMLR records as appearing at AISTATS in 2010 ^[1]. Rather than proposing a new training algorithm, the work asks why conventional deep multi-layer networks are hard to train in the first place, and answers with a rule for setting the initial scale of the weights ^[2].

How it works

The analysis follows the signal as it travels through the network: how activations and gradients vary across layers, and how they change during training ^[1]. The working idea is that training may be harder when the singular values of the Jacobian associated with each layer are far from 1 ^[1]. A singular value above 1 means the layer magnifies whatever passes through it; a value below 1 means it shrinks it. Stack many such layers and either tendency compounds.

A small mental picture helps. Imagine a network with many layers in which every layer scales its input up by a small factor; by the last layer the numbers are enormous. If instead every layer scales its input down a little, the signal reaching the last layer is nearly nothing, and the gradients flowing back are just as weak. Xavier initialization sets the scale of each layer's starting weights in proportion to the size of that layer, so that the signal is neither systematically amplified nor systematically shrunk on the way forward or on the way back, which keeps the layer Jacobians close to the well-behaved case ^[1].

The paper also looks at the choice of activation function. It argues that the logistic sigmoid is unsuited for deep networks with random initialization, because its mean value can drive the top hidden layer into saturation ^[1]. A saturated unit responds almost identically to very different inputs, so learning stalls.

Where it comes from

The paper appeared at AISTATS in 2010 ^[1]. Its abstract notes that all the successful deep-network experimental results since 2006 were obtained with new initialization or training mechanisms ^[1], which frames initialization as one of the levers that made deep networks work at all. The fuller text says the analysis investigates the effects of activation-function choices and initialization procedures on activations and gradients across layers and training iterations ^[2]. It also treats unsupervised pre-training as a particular form of initialization with a drastic impact, and explains that the authors set that route aside in order to analyse what goes wrong with conventional deep multi-layer networks ^[2].

Where it is used today

Initialization is now treated as part of designing a network rather than an afterthought, and the question the paper raised — whether signal scale survives many layers — is a standard one to ask when a deep model will not train. The paper's framing, that how you start the weights can matter as much as how you update them, is why the scheme is usually introduced alongside the basic building blocks of a deep network.

Limits

The paper's target is the conventional deep multi-layer network trained from a random start, not the pre-training route it explicitly sets aside ^[2]. Its diagnosis is stated in terms of the singular values of each layer's Jacobian ^[1], and its conclusions about activation functions belong to the same analysis: the logistic sigmoid is called out as a poor match for deep randomly initialized networks ^[1]. Initialization is a statement about how training begins, arrived at by looking at activations and gradients across layers and over training iterations ^[1]; it does not by itself replace the other choices a deep network requires.