We present a Part-based R-CNN model for fine-grained category detection. Our approach combines the power of deep convolutional neural networks with explicit part-level reasoning to achieve accurate fine-grained recognition. Given an image, we first use R-CNN to detect the whole object and its semantic parts (e.g., head, body, legs for birds). We then extract CNN features from each detected part region and combine them through a geometric constraint model that encodes the expected spatial relationships between parts. Experiments on CUB-200-2011 demonstrate that our method achieves state-of-the-art fine-grained recognition accuracy of 73.9%, significantly outperforming whole-image CNN features and prior part-based methods.

Fine-grained visual categorization aims to distinguish between visually similar sub-categories within a general category, such as differentiating between species of birds, breeds of dogs, or models of cars. This task is particularly challenging because inter-class differences are subtle and localized to specific parts of the object, while intra-class variation due to pose, viewpoint, and illumination can be substantial. Standard whole-image classification approaches, even those using powerful CNN features, may not focus on the discriminative parts that distinguish fine-grained categories.

Prior work on fine-grained recognition has explored part-based models using deformable part models (DPM) and poselets, which explicitly model object parts and their spatial configurations. However, these methods rely on hand-crafted features (HOG) that lack the representational power of deep features. Meanwhile, R-CNN has demonstrated that region-based CNN features can achieve excellent object detection performance. We bridge these two lines of research by applying the R-CNN framework to detect both the whole object and its semantic parts, then combining part-level CNN features with geometric reasoning for fine-grained recognition.

The Part-based R-CNN pipeline consists of three stages. In the detection stage, we train separate R-CNN detectors for the whole object and for each semantic part (e.g., head, body, legs, tail for birds). Each detector uses Selective Search proposals followed by CNN feature extraction and SVM classification. In the feature extraction stage, CNN features (from the fc7 layer) are extracted from the detected bounding boxes of the whole object and each part. In the classification stage, the concatenated part features, augmented with geometric features encoding the relative positions and scales of detected parts, are fed into a final classifier for fine-grained category prediction.

The geometric constraint model encodes the expected spatial configuration of parts relative to the whole object. For each pair of parts, we compute features including relative position (normalized by object bounding box), relative scale, and overlap ratio. These geometric features capture the characteristic pose and layout of each species: for example, the relative position of a bird's head to its body differs systematically between perching and flying poses. The final feature representation concatenates whole-object CNN features, per-part CNN features, and the geometric features, producing a rich, multi-faceted descriptor that captures both local appearance and global structure.

We evaluate Part R-CNN on the CUB-200-2011 benchmark (200 bird species, 11,788 images). Using ground-truth bounding boxes and part annotations at both training and test time, Part R-CNN achieves 73.9% classification accuracy, outperforming whole-image CNN features (65.0%), DPM-based part models (51.0%), and the previous state-of-the-art Poselets + CNN (68.0%). In the more realistic setting where only the bounding box is provided at test time, Part R-CNN achieves 68.7% accuracy using detected parts, still significantly above whole-image baselines. The contribution of part features is most significant for species that differ primarily in head markings or tail patterns, where whole-image features fail to capture the relevant details.

Component analysis reveals the relative importance of each feature type. Whole-object CNN features alone achieve 65.0%. Adding head part features boosts accuracy to 70.2% (+5.2%), confirming that head region is the most discriminative part for bird species recognition. Adding body and leg features further improves to 72.8%. The geometric features contribute an additional 1.1%, bringing the total to 73.9%. We note that the marginal contribution of geometric features is relatively small, suggesting that CNN part features already implicitly encode some positional information through the detected part regions.

We have presented Part-based R-CNNs, an approach that combines deep CNN features with explicit part detection and geometric reasoning for fine-grained visual categorization. Our results on CUB-200-2011 demonstrate that part-level CNN features provide substantial improvements over whole-image features, with the head region being the most discriminative part for bird species recognition. The approach establishes a strong baseline for fine-grained recognition and highlights the importance of focusing computational resources on the most informative regions of the image rather than processing the entire image uniformly. Future work will explore end-to-end training of part detection and classification, as well as learning part detectors without explicit part annotations.