Building High-level Features Using Large Scale Unsupervised Learning

The problem

For years, image recognition depended on two expensive things: labelled data and hand-designed features. Labelling is costly — a benchmark with 20,000 object categories needs humans to look at millions of images and tag them — and hand-designed features such as edges, corners and texture statistics can only encode what researchers already believed was useful.

The paper, presented at ICML 2012 in Edinburgh, asked a narrower but harder question: given nothing but raw unlabelled images, can a single large network discover high-level features by itself? The authors were Quoc V. Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S. Corrado, Jeff Dean and Andrew Y. Ng^[1].

How the network was built

The model was a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization^[1]. Each piece has a job. An autoencoder is trained to reproduce its own input at the output layer, with information passing through a narrower middle layer; to do that well it must keep the useful structure of the input and discard noise. A sparse autoencoder adds pressure toward keeping only a few units active at a time, so different units specialise on different patterns. In a locally connected layer each unit sees only a small patch of the layer below and shares its weights with units looking at other patches — a good match for images, where the same edge or texture can appear anywhere. Pooling merges the responses of neighbouring units so a feature still fires when it shifts by a pixel or two. Local contrast normalization rescales each unit's response by the contrast of its neighbourhood, so the same pattern is recognised under different lighting.

The training data were 10 million 200×200 pixel images downloaded from the Internet^[1]. None of them carried labels.

Training at scale

Scale was the difficulty. The network had 1 billion connections, and the authors trained it with model parallelism and asynchronous stochastic gradient descent (SGD) on a cluster of 1,000 machines with 16,000 cores, for three days^[1]. Model parallelism means splitting the network itself across machines, so a single layer's activations live in different places and must be communicated. Asynchronous SGD means many workers compute gradients on different small batches at once and update a shared set of parameters without waiting for one another; waiting at every step would leave most of the cluster idle. Gradient descent itself is the ordinary procedure of nudging parameters in the direction that lowers the error; "stochastic" means each step uses a small sample rather than the whole dataset.

What the network learned

The result the authors highlighted was that it was possible to train a face detector without labelling any image as containing a face or not^[1]. The network was never told what a face is; a unit in it nevertheless responded strongly to faces. Control experiments showed the detector was robust to translation, scaling and out-of-plane rotation^[1] — the face could move around the frame, sit nearer or farther from the camera, or turn away, and the unit still fired. The same network turned out to be sensitive to other high-level concepts such as cat faces and human bodies^[1].

The features were also usable downstream. Starting from the learned features, the network reached 15.8% accuracy in recognising 20,000 object categories from ImageNet, a 70% relative improvement over the previous state of the art^[1]. With 20,000 categories, guessing is hopeless, so 15.8% is far from a solved problem — but gaining that much over the best previous system, with no labels used to build the features themselves, is what made the paper hard to ignore.

Why it mattered

The Google Research publication page lists the paper as an ICML 2012 publication by the same eight authors^[2]. The paper offers two things at once. On the learning side, raw unlabelled pixels turned out to be enough for a usable high-level feature to emerge, with the face detector as the clearest example. On the engineering side, a model of 1 billion connections trained in three days was only possible by spreading the computation over a thousand machines, so model parallelism and asynchronous updates were not incidental details but the condition for the experiment to exist. The face detector is a clean piece of evidence that labels are not always required to obtain a useful high-level feature.