Learning representations by back-propagating errors

The problem

The paper was published in Nature, volume 323, pages 533–536, in the issue of 9 October 1986. Its authors were David E. Rumelhart and Ronald J. Williams of the Institute for Cognitive Science at the University of California, San Diego, and Geoffrey E. Hinton of the Department of Computer Science at Carnegie-Mellon University.^[1][2]

The problem it addressed was learning in networks with hidden units. A learning rule is easy to find when inputs connect directly to outputs. It becomes much harder when there are hidden units, because the task does not say what their states should be. In the perceptrons of the time, the intermediate “feature analysers” were fixed by hand and did not learn.^[1]

The method

Back-propagation repeatedly adjusts the connection weights so as to minimise the difference between the actual output vector and the desired one. Each unit applies a logistic, S-shaped function to its weighted input. A forward pass computes the state of every unit. A backward pass then uses the chain rule to compute the error derivative for every weight. Gradient descent, with an acceleration term, then updates the weights.^[1]

The important part is the backward pass. It gives a way to assign responsibility for an error to hidden units that are never told directly what they should output. The rest of the method is built from calculus and gradient descent.

Results

The paper tested the procedure on small problems. In the first, a network with two hidden units learned to detect mirror symmetry in 6-bit input vectors, after 1,425 sweeps through the 64 possible inputs.^[1]

In the second, a five-layer network learned two isomorphic family trees, one English and one Italian, from 100 of the 104 possible relationship triples. Isomorphic means the two trees have the same structure with different names. The network answered the four held-out triples correctly, and its hidden units learned features such as nationality, generation and branch of the family.^[1]

The authors were open about the limits. They noted that the error surface may contain local minima, that variants of the procedure were found independently by David Parker and by Yann Le Cun, and that the method in its current form is not a plausible model of learning in brains.^[1]

Why it mattered

The paper’s main claim is that hidden units come to represent important features of the task domain. The authors argued that this ability to create useful new features distinguishes back-propagation from earlier, simpler methods such as the perceptron-convergence procedure.^[1] For a newcomer, the significance is that a network which can build its own intermediate representations is not limited to tasks where a person has already chosen those representations by hand.

What followed

A longer treatment by the same authors, “Learning internal representations by error propagation”, appeared in 1986 in volume 1 of Parallel Distributed Processing (MIT Press), edited by Rumelhart and McClelland.^[2]

ACM’s 2018 Turing Award citation credits the work with showing that backpropagation lets neural networks discover their own internal representations of data. The citation says the algorithm is standard in most neural networks today.^[3] The method it describes is the basis of the backpropagation entry in this archive, and of the later work on convolutional networks that trains with it.