Object detection performance, as measured on the canonical PASCAL VOC dataset, has plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012 — achieving a mAP of 53.3%. Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost. Since we combine region proposals with CNNs, we call our method R-CNN: Regions with CNN features.

Features matter. The last decade of progress on various visual recognition tasks has been based considerably on the use of SIFT and HOG. But performance on canonical recognition tasks has plateaued in recent years. CNNs saw heavy use in the 1990s, but then fell out of fashion with the rise of support vector machines. In 2012, Krizhevsky et al. reignited interest in CNNs by showing substantially higher image classification accuracy on the ImageNet Large Scale Visual Recognition Challenge. The question then became: to what extent do the CNN classification results on ImageNet generalize to object detection on PASCAL VOC?

We answer this question by bridging the gap between image classification and object detection. This paper is the first to show that a CNN can lead to dramatically higher object detection performance on PASCAL VOC as compared to systems based on simpler HOG-like features. To achieve this result, we focus on two problems: localizing objects with a deep network and training a high-capacity model with only a small quantity of annotated detection data. Unlike image classification, detection requires localizing (potentially many) objects within an image. One approach frames localization as a regression problem. An alternative is to build a sliding-window detector. We instead adopt the "recognition using regions" paradigm, generating category-independent region proposals and then evaluating a CNN on each proposal.

While R-CNN is agnostic to the particular region proposal method, we use selective search to enable a controlled comparison with prior detection work. Selective search generates approximately 2,000 category-independent region proposals per image by hierarchically grouping similar regions based on color, texture, size, and fill compatibility. Each proposal defines a tight bounding box around a potentially interesting region, providing a manageable set of candidate locations for the subsequent CNN evaluation.

Features are computed by forward propagating a mean-subtracted 227x227 RGB image through five convolutional layers and two fully connected layers, resulting in a 4096-dimensional feature vector for each region proposal. Regardless of the size or aspect ratio of the candidate region, we warp all pixels in a tight bounding box around it to the required size using affine image warping. Prior to warping, we dilate the tight bounding box so that at the warped size there are exactly p = 16 pixels of warped image context around the original box. This context padding was shown to be important in preliminary experiments.

Supervised pre-training: The CNN is first pre-trained on the ILSVRC 2012 classification dataset (~1.2 million labeled images, 1000 classes). Domain-specific fine-tuning: To adapt the CNN to the new task (detection) and domain (warped proposal windows), we continue SGD training using only warped region proposals. The ImageNet-specific 1000-way classification layer is replaced with a randomly initialized (N+1)-way classification layer (N object classes plus background). We treat all region proposals with IoU overlap >= 0.5 with a ground-truth box as positives for that class, and the rest as negatives. The SGD learning rate is started at 0.001 (1/10th of the initial pre-training rate), and each mini-batch is constructed from 32 positive windows and 96 background windows.

After fine-tuning, we train one linear SVM per class using the CNN features. The overlap threshold of 0.3 was selected by grid search over {0, 0.1, ..., 0.5} on a validation set. Hard negative mining is employed: after an initial SVM is trained, the hardest false positives are collected and the SVM is retrained. For bounding-box regression, inspired by the deformable parts model (DPM), we train a linear regression model from pool5 features to predict new detection windows. This simple post-processing step boosts mAP by 3 to 4 points by correcting mislocalized detections.

On PASCAL VOC 2010, R-CNN achieves 53.7% mAP, compared to 35.1% for the UVA system using identical region proposals but spatial pyramid and bag-of-visual-words features. This 18.6 percentage point improvement (53% relative) demonstrates the power of CNN features over traditional representations. On the 200-class ILSVRC 2013 detection dataset, R-CNN achieves 31.4% mAP, significantly ahead of the second-best result of 24.3% from OverFeat. Furthermore, using the deeper VGGNet (16-layer) architecture as the feature extractor, R-CNN achieves 66.0% mAP on VOC 2007 (up from 58.5% with the original AlexNet-like architecture), though at 7x longer computation time.

A layer-by-layer analysis reveals that much of the CNN's representational power comes from its convolutional layers, rather than the larger fully connected layers. Using only pool5 features (6% of CNN parameters) already achieves reasonable performance, suggesting the convolutional filters learn generic visual patterns. Fine-tuning increases mAP by 8.0 percentage points — critically, the largest improvements occur at fc6 and fc7 layers, indicating that "pool5 features learned from ImageNet are general and that most of the improvement is gained from learning domain-specific non-linear classifiers on top of them." An error analysis using the Hoiem et al. tool reveals that significantly more R-CNN errors result from poor localization rather than confusion with background or other object classes, confirming that CNN features are highly discriminative.

We proposed a simple and effective approach for object detection — R-CNN — that applies a high-capacity CNN to bottom-up region proposals. It is significant that we achieved these results by using a combination of classical tools from computer vision and deep learning (bottom-up region proposals and convolutional neural networks). The supervised pre-training / domain-specific fine-tuning paradigm proved "highly effective for a variety of data-scarce vision problems." We believe these findings will extend to other visual recognition tasks, making it possible to leverage large-scale CNN training for problems where only limited labeled data is available.