AlexNet

What AlexNet is

AlexNet is the deep convolutional neural network described in the paper "ImageNet Classification with Deep Convolutional Neural Networks" by Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton of the University of Toronto, published in the NIPS 2012 proceedings^[1]. Its job was to look at a photograph and say which of a thousand object categories it shows, and it learned that mapping directly from labelled examples.

The architecture has eight learned layers: five convolutional layers, some of them followed by max-pooling, and three fully connected layers ending in a 1,000-way softmax^[1]. In total the network holds 60 million parameters and 650,000 neurons^[1]. A convolutional layer slides small filters across the image and responds to local patterns; stacking such layers lets later ones combine simple patterns into larger structures. The final softmax turns the last layer's scores into a probability for each of the thousand classes.

How it was made practical

Two choices made a network of this size trainable. The first was the use of ReLU units, which the authors showed train several times faster than equivalent tanh units^[1]. The second was a GPU implementation of convolution: training took five to six days on two NVIDIA GTX 580 3 GB GPUs, with the network split across the two GPUs^[1].

A network with 60 million parameters can memorise its training set rather than generalise. To reduce overfitting the authors used data augmentation, feeding the network random 224×224 crops and horizontal reflections of the training images, and dropout with probability 0.5 in the first two fully connected layers^[1]. Dropout switches units off at random during training so that the network cannot lean on any single pathway.

Results

On ILSVRC-2010 test data the network achieved top-1 and top-5 error rates of 37.5% and 17.0%, considerably better than the previous state of the art^[1]. Top-5 error counts a prediction as wrong if the correct label is missing from the model's five highest-scoring guesses.

Entered in ILSVRC2012 as team "SuperVision", it won the classification task with a top-5 error of 15.3%; the next-best entry, the University of Tokyo team "ISI", scored 26.2% with methods based on hand-designed features^[2]. SuperVision also won the ILSVRC2012 localization task, which asks where in the image the named object sits, with 33.5% error against about 50.0% for the next team, Oxford's VGG group^[2].

The authors also probed what was doing the work. They found that depth mattered: removing any single convolutional layer made results worse^[1].

Why it mattered

The organisers' review of the challenge noted that the 15.3% against 26.2% margin showed that a large neural network trained on a large labelled dataset with GPUs could far outperform the feature-based methods of the other entries^[3]. The other competitors mostly built descriptors of image features by hand and fed those to a classifier; AlexNet learned its own features from data instead, and the gap between the two approaches was large enough to be hard to read as a matter of tuning.

The combination that produced the result — a large labelled dataset, a deep convolutional network, ReLU units, dropout and GPU training — became the pattern that later image recognition work started from, and AlexNet is usually the point from which that line of work is counted.