Google Brain network learns cat and face detectors from unlabeled video

Background

The paper Building high-level features using large scale unsupervised learning was listed by Google Research as an ICML 2012 paper by Quoc Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg Corrado, Jeff Dean and Andrew Ng ^[1]. arXiv records its first submission on 29 December 2011 and a revision dated 12 July 2012 ^[2]. The work examined whether a large network trained on raw images could develop useful high-level feature detectors without being given labels for the concepts it was expected to detect.

How the network was built

The authors trained a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on 10 million 200x200 pixel images downloaded from the Internet ^[1][2]. A sparse autoencoder is a network trained to reconstruct its input through a constrained representation; the sparsity constraint encourages only a small number of units to be active at once. Pooling and local contrast normalization are operations that make the representation less sensitive to small changes in position and contrast. The phrase locally connected means that units see local image patches rather than the whole image at every connection. The model had 1 billion connections and was trained with model parallelism and asynchronous SGD on a cluster of 1,000 machines with 16,000 cores for three days ^[1][2].

What it learned

The authors report that it was possible to train a face detector without labelling images as containing a face or not ^[1][2]. Control experiments showed that the feature detector was robust to translation, scaling and out-of-plane rotation ^[1][2]. The same network was also sensitive to cat faces and human bodies ^[1][2]. In other words, the useful features were not supplied as labels; they emerged from training on the image collection. The reported findings include both: a network trained on unlabeled images developed detectors for high-level objects.

Results on ImageNet

Starting from the learned features, the network obtained 15.8% accuracy in recognising 20,000 object categories from ImageNet, a 70% relative improvement over the previous state of the art ^[1][2]. This result is separate from the face-detector demonstration: the same kind of learned representation was used as the starting point for a large object-recognition task. The comparison is a relative improvement, not an absolute accuracy of 70%.

Why the experiment mattered

The experiment combined a deep, locally connected sparse autoencoder with 10 million unlabeled images and distributed training on 1,000 machines ^[1][2]. It showed that a network could learn face-selective and cat-selective features without labels for those categories, and that the resulting features could support large-scale object recognition ^[1][2]. The paper's public record is also clear: Google Research lists it as an ICML 2012 paper, and arXiv records the December 2011 submission and the July 2012 revision ^[1][2]. For readers following deep learning, the work is a large-scale demonstration that unsupervised feature learning could produce detectors for high-level visual concepts and then be reused for recognition.