Connectionism revives with backpropagation and the PDP volumes

Background

Connectionism is the approach that explains intelligence with networks of simple, neurone-like units wired together, rather than with symbol manipulation alone. In the 1980s the idea came back into view in what is usually called the second wave of connectionism^[1]. For that wave to be credible, two questions had to be answered: could such a network learn anything useful once extra layers of units sat between input and output, and could it be trained in practice? Minsky and Papert had argued a pessimistic case about learning in multilayer machines, and their book *Perceptrons* was the reference point anyone working on the question had to face^[2].

The paper

On 9 October 1986 Nature published "Learning representations by back-propagating errors" by David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, in volume 323, pages 533 to 536^[3]. Rumelhart and Williams were then at the Institute for Cognitive Science at the University of California, San Diego; Hinton was in the Department of Computer Science at Carnegie-Mellon University^[3].

How the method works

As the abstract describes it, back-propagation is a learning procedure for networks of neurone-like units. The network repeatedly adjusts the weights of the connections between its units so as to minimise a measure of the difference between the output vector it actually produces and the output vector that is wanted^[3]. In effect the error at the output is used to work out how much each connection contributed to it, and each weight is nudged in the direction that makes the error smaller. The units are not linear, and the discovery of non-linear activation functions is credited with enabling the second wave as a whole^[1].

Hidden units and representations

The claim that mattered most concerns what happens inside the network. Internal hidden units, the abstract says, come to represent important features of the task domain, and the ability to create useful new features is what distinguishes back-propagation from earlier, simpler methods such as the perceptron-convergence procedure^[3]. A hidden unit is never told directly what it should respond to; its weights are shaped only by the effect its output has on the final error. Features that help reduce the error therefore get built into the network even though nobody specified them in advance.

The PDP volumes

The 1986 paper did not appear on its own. Wikipedia lists it together with the two-volume *Parallel Distributed Processing* book of 1987 as the key publications of the second wave of connectionism^[1]. The paper's own references include Rumelhart, Hinton and Williams in *Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations*, published by MIT in 1986, pages 318 to 362^[3]. The PDP work introduced hidden layers and sigmoid activation functions as improvements on the simple perceptron^[1].

Settling accounts with *Perceptrons*

With connectionism revived in the late 80s, the PDP researcher David Rumelhart and colleagues returned to *Perceptrons*. In a 1986 report they claimed to have overcome the problems Minsky and Papert had presented, and said that Minsky and Papert's pessimism about learning in multilayer machines was misplaced^[2]. The disagreement left a mark on the book itself: the 1988 expanded edition of *Perceptrons* added a prologue and an epilogue discussing the revival of neural networks in the 1980s^[2].

Why it matters

Wikipedia attributes the popularisation of back-propagation to the October 1986 paper^[1]. What changed was not only a training algorithm but the picture of what a network is for. A multilayer network trained by back-propagation is not just a classifier fitted to fixed inputs; it builds its own internal features, and those features can then be used for other tasks. That is why the paper and the PDP volumes are treated as the opening of the second wave, and why the argument over multilayer learning that had dominated the previous decade could be set aside^[1][2].