Rich feature hierarchies for accurate object detection and semantic segmentation

On 11 November 2013, Ross Girshick, Jeff Donahue, Trevor Darrell and Jitendra Malik posted a paper to arXiv titled "Rich feature hierarchies for accurate object detection and semantic segmentation"^[1]. It described a detection method its authors called R-CNN, short for regions with convolutional neural networks, because the method combines region proposals with convolutional neural networks^[1].

A benchmark that had stopped improving

Object detection is the task of telling a computer not only what is in a photograph but where it is: the system has to draw a box around each object of interest. Progress is usually measured on PASCAL VOC, a public benchmark of photographs annotated with such boxes. The standard score is mean average precision (mAP), which combines, over all object classes, how many of the system's detections were correct and how much of the annotated truth it managed to find; a higher number means better detection.

The authors noted that detection performance on PASCAL VOC had plateaued, and that the best methods at the time were complex ensemble systems that typically combine multiple low-level image features with high-level context^[1]. In other words, the leading systems were elaborate assemblies of hand-designed descriptors, and each further improvement was small.

Region proposals meet deep networks

Instead of designing image features by hand, the authors proposed a detection algorithm that applied high-capacity convolutional neural networks to bottom-up region proposals in order to localise and segment objects^[1].

The recipe has two parts. A region proposal stage works bottom-up from the image itself, grouping pixels and edges into a few thousand candidate boxes that plausibly contain an object, without yet deciding what the object is. A convolutional neural network (CNN) then looks at each candidate. A CNN is a stack of learned filters that builds a hierarchy of features: the early layers respond to simple patterns such as edges and colour boundaries, the later layers respond to object parts and then to whole objects. Running such a network on a candidate box therefore yields a rich feature hierarchy for that region rather than a short list of hand-designed measurements, and a classifier on top of those features decides which class, if any, the region contains. This is the sense in which the paper's title speaks of rich feature hierarchies.

Training when labelled data is scarce

Detection datasets are small in comparison with the image collections used for general classification, because annotating a bounding box by hand is expensive. The authors said that when labelled training data is scarce, supervised pre-training for an auxiliary task followed by domain-specific fine-tuning yields a significant performance boost^[1]. Concretely, the network is first trained on a large auxiliary task where labels are plentiful, so that its filters learn generally useful visual features, and is then fine-tuned on the smaller, domain-specific detection data so that those features adapt to the objects and images at hand.

Results

The method improved mean average precision by more than 30% relative to the previous best result on VOC 2012, reaching a mAP of 53.3%^[1]. Coming after a period in which the benchmark had plateaued, a jump of that size was the paper's headline finding.

Why it mattered

The result made the case for two ideas. The first is that detection features can be learned by a deep network instead of assembled from hand-designed descriptors, since the work applied high-capacity CNNs to region proposals in order to localise and segment objects^[1]. The second is the transfer recipe: pre-train on an auxiliary task, then fine-tune on the target domain, which the authors reported gives a significant boost when labelled data is scarce^[1]. Together these showed that the plateau described in the paper was not a limit of the task itself. The same regions and features were used for both localisation and segmentation, as the title's mention of semantic segmentation indicates. And R-CNN gave the line of work a name: region proposals plus convolutional networks^[1].