Gradient-Based Learning Applied to Document Recognition

The problem

The review appeared in Proceedings of the IEEE, volume 86, issue 11, pages 2278–2324, in November 1998. Its authors were Yann LeCun, Léon Bottou, Yoshua Bengio and Patrick Haffner of the Speech and Image Processing Services Research Laboratory at AT&T Labs-Research in Red Bank, New Jersey. Bengio was also at the Université de Montréal.^[1][2]

Many recognition systems of the time depended on hand-designed heuristics and hand-crafted feature extraction, which had to be built for each task. The review’s main message is that better pattern-recognition systems can be built by relying more on automatic learning and less on hand-designed heuristics. Hand-crafted feature extraction can be replaced by learning machines that work directly on pixel images.^[1]

The review

The paper reviews many methods for handwritten character recognition and compares them on a standard handwritten-digit task. It shows that convolutional neural networks outperform all the other techniques tested.^[1] It also treats reading as a whole system, with segmentation, recognition and language modelling as parts of one pipeline, rather than as isolated character classifiers.

LeNet-5

The paper describes LeNet-5, a convolutional network with seven layers, not counting the 32x32 input. It alternates convolution and sub-sampling layers, labelled C1, S2, C3 and S4 in the paper, then has a convolutional layer C5 with 120 maps, a layer F6 with 84 units, and 10 output units. The network has 340,908 connections but only about 60,000 trainable parameters, because many connections share weights.^[2]

The MNIST dataset

The paper introduced the MNIST (“Modified NIST”) dataset, built by mixing NIST Special Databases 1 and 3. It has 60,000 training images and, in the paper’s experiments, 10,000 test images of handwritten digits. The digits were size-normalised to 20x20 and centred in a 28x28 field.^[1][2]

MNIST later became a familiar benchmark in machine learning teaching and research.

Results

LeNet-5 reached 0.95% test error on MNIST. Adding 540,000 artificially distorted copies of the training images lowered the error to 0.8%.^[1]

Graph transformer networks and check reading

The paper introduces Graph Transformer Networks. These allow a multi-module recognition system, with modules for segmentation, recognition and language modelling, to be trained globally with gradient-based methods. The aim is to optimise one overall performance measure for the whole system.^[1]

It also describes a bank-check reading system that combines convolutional character recognisers with global training. The system was deployed commercially and read several million checks per day.^[1]

Why it mattered

The review makes the case for learning features from data instead of designing them by hand. That case is the thread connecting the earlier zip-code work and the convolutional network design, which this paper summarised and extended. Its examples show the approach working on a standard benchmark and on a commercial system, and it is a useful reference for the history of convolutional networks.