Backpropagation Applied to Handwritten Zip Code Recognition

The problem

The paper appeared in Neural Computation, volume 1, issue 4, pages 541–551, in 1989. It addresses a practical task: reading the handwritten zip codes on U.S. mail. Seven authors from AT&T Bell Laboratories in Holmdel, New Jersey wrote it: Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel.^[1][2]

The group’s earlier system had hand-chosen first layers. This paper builds a single network that learns the whole recognition task, from the normalised image to the final class. Every connection is learned with backpropagation, and the network is fed images rather than hand-made feature vectors.^[2]

Data and architecture

The data were 9,298 segmented handwritten digits from zip codes on U.S. mail passing through the Buffalo, New York post office, supplied through the U.S. Postal Service. Of these, 7,291 were used for training and 2,007 for testing. Each digit was normalised to a 16x16 pixel image.^[2]

The network has three hidden layers. The first has 12 feature maps of 8x8 units. Each unit sees a 5x5 patch of the image, and all units in one map share the same 25 weights. The authors describe this as a non-linear, sub-sampled convolution with a 5x5 kernel. Next come 12 maps of 4x4 units, then 30 fully connected units and 10 outputs.^[2]

Weight sharing keeps the network small in terms of what has to be learned. It has 1,256 units and 64,660 connections, but only 9,760 independent parameters.^[1]

Training and results

Training made 23 passes through the training set, 167,693 pattern presentations in total, with stochastic gradient updates after each pattern. It ran on a SUN-4/260 workstation and took about 3 days.^[2]

The network reached 0.14% error on the training set and 5.0% on the test set, which is 102 mistakes. For comparison, a fully connected network with one hidden layer of 40 units had 8.1% test error. The authors judged the result to be at the state of the art in digit recognition.^[2]

The trained network ran on an off-the-shelf AT&T DSP-32C digital signal processor board. It classified 10 to 12 digits per second including image acquisition, and more than 30 per second on normalised digits.^[2]

Why it mattered

The paper shows the convolutional structure and backpropagation training working together on a real-world problem, with the data and error figures reported in full. The authors attribute the success to what they call constrained backpropagation. The first two layers are constrained to be convolutional, but the network learns the kernel coefficients itself. Some of the learned kernels resembled the feature detectors that Hubel and Wiesel (1962) had found in biological vision systems.^[2]

What followed

This is the paper that introduces the convolutional neural network, which has its own entry in this archive. Its design was later extended in LeNet-5, described in a separate paper. The network also shows what a learned system can do without hand-designed early layers, a question that the later convolutional work continued to pursue.