Parallel Distributed Processing (PDP volumes)

The book and its two volumes

*Parallel Distributed Processing: Explorations in the Microstructure of Cognition* was published by MIT Press as a two-volume work. Volume 1, *Foundations*, is credited to David E. Rumelhart, James L. McClelland and the PDP Research Group; Volume 2, *Psychological and Biological Models*, lists McClelland first and Rumelhart second, again together with the PDP Research Group.^[1][3] The publisher's catalogue entry for the first volume and the Stanford library record both give 1986 as the publication year,^[2][3] while some accounts describe the work as a 1987 two-volume book.^[1] The gap is a small bibliographic one; the substance of the work is the same either way.

The picture of the mind it argues for

The publisher's summary of the book states the authors' theory directly: the mind is composed of a great number of elementary units connected in a neural network, and mental processes are the interactions between these units, carried out in parallel rather than as sequential operations.^[3] "In parallel" is the load-bearing phrase — many units are active and influencing one another at the same moment, instead of a single processor executing one instruction after another. The general principle behind this, described as the central principle of connectionism, is that mental phenomena can be described by interconnected networks of simple and often uniform units.^[1] In such a network, memory and learning are treated as modifications of the connection weights, generally represented as an n×m matrix.^[1] Each entry in that matrix says how strongly the activity of one unit pushes or pulls another; learning means changing those numbers rather than filing items away in a separate store.

Departures from the simple perceptron

A perceptron is a single unit that adds up weighted inputs and fires when the total crosses a threshold. The book introduced improvements to this idea that included intermediate processors, or hidden layers, alongside the input and output units.^[1] It also describes the use of a sigmoid activation function in place of an all-or-nothing function.^[1] The difference matters once units are stacked in layers: an all-or-nothing unit switches abruptly between off and on, whereas a sigmoid changes smoothly, so small adjustments to a weight produce small changes in behaviour — which is what makes gradual adjustment of a whole network possible. The discovery of non-linear activation functions is in turn credited with enabling the second wave of connectionism.^[1]

Backpropagation, and where the book sits

The popularisation of backpropagation — the procedure that works out how each weight should change by sending the error backwards through the network, layer by layer — is attributed to a separate 1986 paper by Rumelhart, Hinton and Williams, "Learning representations by back-propagating errors", published in October 1986.^[1] The PDP volumes and that paper are together listed as key publications of the second wave of connectionism.^[1] The book supplied the framework and vocabulary; the paper supplied the widely used learning rule.

Recognition

In 2002, Rumelhart and McClelland were awarded the University of Louisville Grawemeyer Award for Psychology for their work on the cognitive framework called parallel distributed processing and the concept of connectionism.^[2]