ImageNet Classification with Deep Convolutional Neural Networks

The problem

The task was to classify the 1.2 million high-resolution training images of the ImageNet ILSVRC-2010 contest into 1,000 classes. Such a large and varied dataset demanded a model with great capacity, but a network of that size was hard to train well without overfitting, meaning that it memorises its training examples instead of learning patterns that carry over to new images.^[1]

How it works

A convolutional layer slides small learned filters across an image, so the same pattern can be detected wherever it appears in the picture. The network, known as AlexNet, has eight learned layers: five convolutional layers, some followed by max-pooling, and three fully connected layers that end in a 1,000-way softmax, which turns the outputs into probabilities over the classes. It has 60 million parameters and 650,000 neurons.^[1]

The authors made several choices that made training practical. They used ReLU (rectified linear) units, which they showed train several times faster than equivalent tanh units. They wrote a highly optimised GPU implementation of convolution, and split the network across two GPUs. Training took five to six days on two NVIDIA GTX 580 GPUs with 3 GB of memory each.^[1]

To limit overfitting, the authors enlarged the training set through data augmentation: random 224×224 crops and horizontal reflections, which increased the training data by a factor of 2,048. They also used dropout, which sets each hidden neuron's output to zero with probability 0.5 in the first two fully connected layers. Dropout discourages the network from relying on any single neuron.^[1]

Results

Top-1 error counts a prediction as wrong unless the model's single best guess matches the label. Top-5 error is more lenient: a prediction counts as correct if the label is among the model's five best guesses.

On ILSVRC-2010 test data, the network achieved top-1 and top-5 error rates of 37.5% and 17.0%, which the authors described as considerably better than the previous state of the art.^[1] A variant entered in the ILSVRC-2012 competition achieved a winning top-5 test error of 15.3%, compared with 26.2% for the second-best entry.^[1]

Why it mattered

The authors found that depth mattered. Removing any single convolutional layer, each of which held no more than 1% of the parameters, made results worse.^[1] The result showed that a deep convolutional network trained on a large labelled dataset could far outperform earlier methods, and it gave the field a clear example of what GPU-based training could achieve.

Later influence

Convolutional networks of this kind became the dominant approach to image recognition in the years that followed. The paper is often cited as a starting point for that shift, although the ideas behind it, including convolutional networks and backpropagation, had earlier roots.^[1]