Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

The problem

Object detection asks a system to say what objects are in an image and where they are. At the time of the paper, the authors wrote that state-of-the-art detection networks depended on region proposal algorithms to hypothesise object locations ^[1]. In that setup, a separate procedure first suggests image regions that might contain objects; the detector then classifies and refines those regions. The paper's starting point is that this division of labour makes the proposal step a distinct part of the pipeline, with its own computation, rather than something the detection network does for itself ^[1]. The question the paper addresses is how to produce good region proposals without treating them as an external, separately run stage.

The method

The paper was posted to arXiv on 4 June 2015 by Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun ^[1]. Its central device is a Region Proposal Network that shares full-image convolutional features with the detection network, enabling nearly cost-free region proposals ^[1]. In a convolutional network, filters run over the whole image and produce features that describe local image content; if the proposal network and the detector use the same such features, the proposal step does not need a separate feature-extraction pass. The authors describe the resulting proposals as nearly cost-free ^[1]. The Region Proposal Network and the detection network can be trained end-to-end by sharing convolutional features ^[1]. End-to-end here means the two parts are optimized together through the shared layers rather than trained as unrelated stages; the shared features carry signals from both the proposal task and the detection task. The paper therefore presents a unified network in which proposing regions and detecting objects are coupled through common convolutional computation ^[1].

Results

The unified network allowed a detection system to run at 5 frames per second on a GPU ^[1]. The authors also reported state-of-the-art object detection accuracy on PASCAL VOC 2007, PASCAL VOC 2012 and MS COCO with a single image per scale ^[1]. The speed figure and the accuracy claim are presented together: the same system that shares features for proposals is the one evaluated for detection quality and throughput ^[1]. “Single image per scale” describes the input condition for the reported accuracy; the result was not presented as requiring a stack of images at each scale ^[1].

Why it matters

The paper's importance lies in the coupling it proposes. Region proposals had been a separate algorithmic component that hypothesised object locations before detection ^[1]. By making the proposal network share full-image convolutional features with the detector, the paper turns proposal generation into part of the same convolutional network and makes the proposals nearly cost-free ^[1]. That changes the engineering picture: the detector is no longer a proposal method followed by a classifier, but a unified network trainable end-to-end ^[1]. The reported 5 frames per second on a GPU gives the work its “towards real-time” framing ^[1]. The accuracy results on PASCAL VOC 2007, 2012 and MS COCO show that this integration was not presented as a speed-only trade-off; the authors reported state-of-the-art accuracy at the same time ^[1].