A Learning Algorithm for Boltzmann Machines

The setting

A Learning Algorithm for Boltzmann Machines is a 1985 article by David H. Ackley and Geoffrey E. Hinton of the Computer Science Department at Carnegie-Mellon University, and Terrence J. Sejnowski of the Biophysics Department at Johns Hopkins University. It appeared in Cognitive Science volume 9, pages 147–169 ^[1]. The article opens by noting that evidence about the architecture of the brain and the potential of new VLSI technology had led to a resurgence of interest in connectionist systems ^[1]. In such networks, long-term knowledge is stored as the strengths of the connections between simple neuron-like processing elements ^[1]. The opening therefore sets up the paper's basic picture: a network is made of simple elements and connections, and what the network knows is carried by how strong those connections are.

Why parallel networks

The authors argue that these networks suit tasks such as vision that can be performed efficiently in parallel, because their physical connections sit exactly where processes need to communicate ^[1]. The paper also identifies large constraint satisfaction as one kind of computation for which massively parallel networks are well suited ^[1]. In general terms, a constraint satisfaction problem asks for an arrangement that satisfies a set of conditions at the same time; the paper's claim is that this kind of problem fits a massively parallel network ^[1]. Vision and large constraint satisfaction are thus presented as cases where the organization of a connectionist network matches the demands of the computation ^[1].

The learning rule

Against this background, the paper describes a general learning rule for modifying connection strengths so as to incorporate knowledge about a task domain from examples ^[1]. The rule is presented as general: it is a procedure for changing connection strengths, and knowledge about a task domain enters through examples ^[1]. Because long-term knowledge is stored in the connection strengths, learning in this setting is the adjustment of those strengths ^[1]. The paper's focus is therefore on changing the connections that carry the network's knowledge ^[1].

Reported examples

The article reports simple examples in which the learning algorithm creates internal representations with demonstrably efficient structure ^[1]. The examples are described as simple, and the paper uses them to show that the algorithm can produce internal representations whose structure is demonstrably efficient ^[1]. The paper's claim is stronger than a report that connection strengths change: it reports that the algorithm creates internal representations, and that their structure is demonstrably efficient ^[1].

Why it matters

The paper's importance lies in how it joins several concerns. It begins with connectionist systems that store long-term knowledge in connection strengths between simple neuron-like elements ^[1]. It argues that such networks fit parallel tasks such as vision and large constraint satisfaction because their connections place processes where they need to communicate ^[1]. And it offers a general learning rule that uses examples to modify those connections, with simple demonstrations that the resulting internal representations have efficient structure ^[1]. In doing so, it presents learning as a problem of connection strengths and internal representation, set within an architecture motivated by evidence about the brain and by the potential of VLSI technology ^[1]. These are the elements the paper puts together: a network architecture, a parallel style of computation, a general rule for learning from examples, and a demonstration that learning can create structured internal representations ^[1].