Convolutional neural networks

What it is

A convolutional neural network (CNN) is a neural network designed for images and other grid-like data. Instead of connecting every input to every unit, each unit looks only at a small patch of the layer below. The design combines three ideas, as summarised by LeCun, Bottou, Bengio and Haffner in 1998: local receptive fields, shared weights, and spatial sub-sampling.^[2]

How it works

Imagine a small detector that looks for a short diagonal edge within a patch of, for example, 5x5 pixels. The detector is applied at every position in the image, producing a feature map that shows where the edge appears. Every unit in one feature map uses the same weights, so the map detects the same feature everywhere. Sharing weights in this way is what makes the operation a convolution followed by a non-linearity.^[2]

Weight sharing also sharply reduces the number of free parameters. LeNet-5, from 1998, had 340,908 connections but only about 60,000 trainable parameters. Because shifting the input simply shifts the feature map, the network is robust to shifts and distortions in the image.^[2]

Sub-sampling layers, also called pooling layers, reduce the resolution of the feature maps, so the exact position of a feature matters less. Convolution and sub-sampling layers alternate, and as resolution falls the network uses more feature maps. A final stage combines the detected features into a decision.^[2]

Where it came from

The 1998 paper cites two origins. One is Hubel and Wiesel’s discovery of locally sensitive, orientation-selective neurons in the cat’s visual cortex. The other is Kunihiko Fukushima’s Neocognitron, from 1980, which had a similar layered structure but no globally supervised learning procedure such as backpropagation.^[2]

In 1989 Yann LeCun and colleagues at AT&T Bell Laboratories trained such a constrained network end to end with backpropagation. The task was to read handwritten zip-code digits from U.S. mail.^[1]

Key milestones

The 1989 system reached 5.0% error on the test set. The network had 1,256 units, 64,660 connections and 9,760 independent parameters.^[1]

LeNet-5 reached 0.95% test error on the MNIST handwritten-digit dataset, and 0.8% when it was trained with extra artificially distorted images. A check-reading system built on convolutional networks was deployed commercially and read several million checks per day.^[2]

Where it is used today

In 2012 AlexNet, a much larger convolutional network with five convolutional layers and 60 million parameters, trained on GPUs, won the ImageNet ILSVRC-2012 competition. It had a 15.3% top-5 test error, against 26.2% for the second-best entry.^[3]

ACM’s 2018 Turing Award citation describes convolutional networks as an industry standard in computer vision. It also notes their use in speech recognition, speech synthesis, image synthesis and natural language processing, and in applications such as autonomous driving and medical image analysis.^[4]

Limits

A convolutional network assumes that useful evidence is local: a feature is a pattern in a small neighbourhood, and the same pattern can recur across the image. That assumption suits images well but fits other kinds of data less naturally. Pooling discards exact position, which helps recognition but hurts tasks where precise location matters. Training also needs many labelled examples and substantial computing power.