Boltzmann machine

What it is

A Boltzmann machine is a neural network in which the state of the whole network is described by a single number, its energy. The units are connected to one another, and every way of setting those units on or off — every configuration — has an energy value attached to it. The global energy E takes the same form as in Hopfield networks and Ising models^[1], and the machine is built on the Sherrington–Kirkpatrick spin glass model^[1]. That lineage shapes how the model is used: rather than computing an output step by step, the network behaves like a physical system that settles into low-energy states, and computing means finding those states.

How it works

Two things set the behaviour. First, the units are stochastic: whether a unit turns on is decided at random, with the odds fixed by how much the energy of the whole network would fall or rise if that unit flipped, and by a temperature parameter. Second, the connections carry weights, and the weights are what learning changes.

Temperature supplies a way to search. At high temperature the network wanders among configurations; as temperature falls it stays increasingly in low-energy ones. This is simulated annealing — the technique Hinton was due to speak about in 1983, in the setting of Hopfield networks^[1].

A small example. Take four units and suppose we want the first two to be on together and the last two to be on together. If the weight within each pair is made large and positive, configurations where each pair agrees have much lower energy than configurations where one member of a pair is on and the other off. The energy function is thus a compact way of writing down which combinations of unit states are acceptable — the reason a 1984 technical report called these networks constraint satisfaction networks that learn^[2].

Learning adjusts the weights so that the network gives low energy to the patterns it is shown and higher energy to the configurations it produces on its own, with the difference between the two driving the change. Hinton later recalled that in February or March 1983 he was due to give a talk on simulated annealing in Hopfield networks and had to design a learning algorithm for it; that algorithm became the Boltzmann machine learning algorithm^[1].

Where it came from

The immediate background was Hopfield's 1982 publication, which applied spin glass theory to associative memory^[1]. The Boltzmann machine grew out of that line of work. By Hinton's own account the learning algorithm dates from early 1983, when the talk on simulated annealing forced him to work out how such a network could learn^[1].

Wikipedia's references for the model include Hinton–Sejnowski papers from May 1983 and June 1983, and a 1983 AAAI-83 paper on Boltzmann machines^[1]; the Salk Institute publication database lists Hinton and Sejnowski's "Massively Parallel Architectures for AI: NETL, Thistle and Boltzmann Machines" in the Proceedings of the National Conference on Artificial Intelligence, Washington, D.C., 1983^[2]. A 1984 Carnegie-Mellon University computer science technical report, CMU-CS-84-119, carried the title "Boltzmann Machines: Constraint Satisfaction Networks That Learn", by Hinton, Sejnowski and Ackley^[2].

The full learning procedure was published with Ackley: "A Learning Algorithm for Boltzmann Machines", by Ackley, Hinton and Sejnowski, in Cognitive Science, volume 9, pages 147 to 169, 1985^[1][2]. A 1986 chapter, "Learning and Relearning in Boltzmann Machines", by Hinton and Sejnowski, appeared in McClelland and Rumelhart's Explorations in the Microstructure of Cognition, volume 1, from MIT Press^[2].

Why it matters

Wikipedia attributes the original contribution in applying energy-based models in cognitive science to the papers of Hinton and Sejnowski^[1], and credits the wide spread of Boltzmann machines in the cognitive science and machine learning communities to Hinton, Sejnowski and Yann LeCun^[1]. The lasting move is conceptual: a model can be defined by an energy that scores every possible configuration, and learning can be defined as reshaping that score. Weights become the shape of an energy landscape rather than a pipeline of steps, and what the network has learned is expressed as a set of constraints that low-energy states satisfy^[2].

Limits

The price is computation. Learning compares statistics gathered while the network is shown data with statistics gathered while it runs on its own, and each of those estimates needs many samples of the network's own configurations. Training is therefore slow compared with models whose state can be computed in a single pass, and the sampling has to be guided carefully by the temperature schedule.