A fast learning algorithm for deep belief nets

Geoffrey E. Hinton, Simon Osindero and Yee-Whye Teh published "A fast learning algorithm for deep belief nets" in the journal Neural Computation on 1 July 2006; the article appeared in volume 18, issue 7, pages 1527–1554 ^[1]. Hinton and Osindero were in the Department of Computer Science at the University of Toronto, and Teh at the National University of Singapore ^[1].

A belief net is a probabilistic model in which visible units hold the data and hidden units hold explanations of it, linked by weighted connections, so that the whole network defines a probability distribution over the visible data. More hidden layers let such a network represent richer, hierarchical regularities, but they also make learning harder. In a densely connected net with many hidden layers, inferring the state of the hidden units from the data is difficult because of explaining away: when several hidden causes could each account for what is observed, evidence for one of them lowers the probability of the others, so the hidden units are no longer conditionally independent and have to be inferred jointly. Learning then becomes slow and unreliable. Removing that obstacle is what the paper sets out to do ^[1].

Complementary priors

The authors' move is to change the prior. The abstract says they use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets with many hidden layers ^[1]. Roughly, the prior is chosen to fit the likelihood implied by the weights so that the coupling cancels: when the prior is complementary to the likelihood, the posterior over hidden units factorizes and each unit can be inferred on its own instead of jointly with all the others. Once inference inside a layer becomes easy, the network can be built up one layer at a time.

Learning one layer at a time

Using complementary priors, the paper derives a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided that the top two layers form an undirected associative memory ^[1]. The top two layers are left undirected and act as an associative memory holding the representation learned so far; the layers below are directed and generate the layer above them. The greedy procedure adds and trains one layer at a time, so no single step has to solve the full deep inference problem at once — each new layer only has to model the representation produced by the layers already in place.

Fine-tuning

The greedy pass is only an initialization. The paper uses the fast, greedy algorithm to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm ^[1]. Wake-sleep alternates two passes: in the wake phase the model adjusts its generative connections so as to explain the data it is shown, and in the sleep phase it adjusts its recognition connections so as to invert samples drawn from its own generative model. The contrastive version used here continues from the weights the greedy pass has already found and refines them.

Results

After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels ^[1]. Because the model is generative — it describes the joint distribution of images and labels rather than only a boundary between classes — it can be asked for the most probable label given an image. The paper states that this generative model gives better digit classification than the best discriminative learning algorithms ^[1]. The point is worth stating plainly: a model built to describe the data, not to separate classes, classified digits better than methods trained for classification.

Why it mattered

The paper's contribution is best seen as two connected pieces. The first is a way of making inference tractable in a deep, densely connected network, so that the difficulty of explaining away no longer blocks learning; the second is a workable training schedule built on top of it — a fast greedy pass that grows the network layer by layer, followed by a slower fine-tuning pass that treats the whole network together. Neither piece is presented as an isolated trick: the greedy algorithm supplies the starting point, and the contrastive wake-sleep procedure supplies the polish.

The most portable part of the work is its two-stage shape. A deep model is first given a reasonable set of weights by training its parts one at a time, each part on the representation produced by the parts below it, and only then adjusted as a whole. That ordering turns an intractable joint problem into a sequence of smaller ones, and it is described here clearly enough to be reused with different components. The digit experiment also makes a broader argument: a network that models the joint distribution of inputs and labels carries more information than one that only draws a boundary between classes, and in this case that extra information showed up as better classification ^[1].