Learning Multiple Layers of Features from Tiny Images

The problem

The report is a technical report by Alex Krizhevsky, dated 8 April 2009 ^[1]. Its abstract notes that groups at MIT and NYU had collected a dataset of millions of tiny colour images from the web ^[1]. The report asks how a model can learn useful features from such images. It frames the goal as training a multi-layer generative model that learns to extract meaningful features resembling those found in the human visual cortex ^[1]. The emphasis is therefore on feature learning, not only on classifying raw pixels.

Approach

The report trains a multi-layer generative model ^[1]. It focuses on Restricted Boltzmann Machines and Deep Belief Networks ^[1]. In general terms, a Restricted Boltzmann Machine is a generative model that uses hidden units to represent structure in the input, and a Deep Belief Network stacks multiple layers of such representations. The report says the learned filters of these models are more useful to a classifier than raw pixels ^[1]. To make training feasible, a novel parallelization algorithm distributes the work among multiple machines connected on a network so that training can be done in reasonable time ^[1]. Another key element is pre-training: object recognition is significantly improved by pre-training a layer of features on a large set of unlabeled tiny images ^[1]. In this setup, unlabeled images first help the model form a representation, which is then used in recognition.

Labels

The report says two sets of reliable labels were created ^[1]. CIFAR-10 contains 6000 examples of each of 10 classes ^[1]. CIFAR-100 contains 600 examples of each of 100 non-overlapping classes ^[1]. These labeled sets give the study explicit targets for evaluating the learned features.

Results and significance

The report's stated result is that object recognition is significantly improved by pre-training a layer of features on a large set of unlabeled tiny images ^[1]. It also reports that the filters learned by Restricted Boltzmann Machines and Deep Belief Networks are more useful to a classifier than raw pixels ^[1]. The parallelization algorithm distributes training work across multiple machines on a network, allowing training in reasonable time ^[1]. The report documents both CIFAR-10 and CIFAR-100 labels ^[1]. The University of Toronto CIFAR page asks anyone using the dataset to cite the technical report at the bottom of that page ^[2]. The report is therefore not only a description of a method but also the citation tied to the dataset ^[2]. By bringing together unsupervised feature learning, distributed training, and labeled sets of tiny images, the report connects several concerns in the study of image features.