In object detection, the intersection over union (IoU) threshold is frequently used to define positives and negatives. A fundamental problem exists: training with larger thresholds causes performance degradation despite their intuitive appeal. The authors identify two core issues: overfitting from vanishing positive samples at high thresholds, and inference-time mismatches between detector quality and hypothesis quality. Their solution involves "a sequence of detectors trained with increasing IoU thresholds," where each stage uses the previous stage's output as training input. This Cascade R-CNN achieves state-of-the-art performance on COCO with significant improvements on stricter metrics.

Object detection requires solving two complementary tasks: recognition (distinguishing foreground from background) and localization (assigning accurate bounding boxes). The IoU threshold traditionally set at 0.5 is "a very loose requirement for positives," producing noisy detections that humans would reject as incorrect. A core observation drives the work: "a single detector can only be optimal for a single quality level." This creates the "paradox of high-quality detection" -- training with higher thresholds paradoxically degrades performance due to data scarcity and distribution mismatches.

Prior multi-stage detection approaches differ fundamentally from Cascade R-CNN. Iterative bounding box regression applies "a single regressor iteratively" post-hoc, but this fails because "a regressor trained at u=0.5 is suboptimal for hypotheses of higher IoUs." Integral loss training uses "an ensemble of classifiers trained with the integral loss," but doesn't address the positive sample imbalance problem. The authors distinguish their approach: "the resampling performed by the Cascade R-CNN does not aim to mine hard negatives. Instead, by adjusting bounding boxes, each stage aims to find a good set of close false positives for training the next stage."

Two factors explain the limited progress on high-quality detection. First, "evaluation metrics have historically placed greater emphasis on the low quality detection regime," with many datasets using u=0.5 evaluation, saturating performance at loose quality levels. Second, "the design of high quality object detectors is not a trivial generalization of existing approaches, due to the paradox of high quality detection." Simply increasing training threshold u causes overfitting as positive samples vanish exponentially -- the RPN produces hypotheses "heavily tilted towards low quality" with only 2.9% exceeding IoU=0.7.

Rather than optimizing a single regressor for all quality levels, Cascade R-CNN decomposes regression into "a cascade of specialized regressors" where each stage optimizes for the bounding box distribution from the previous stage. This leverages the observation that "the output IoU of a bounding box regressor is almost always better than its input IoU." Each detection stage t includes classifier h_t and regressor f_t optimized for threshold u^t, trained with combined classification and localization loss. The resampling mechanism guarantees "a positive training set of equivalent size for all detectors," preventing overfitting while enabling specialization.

On MS-COCO 2017, the vanilla Cascade R-CNN with ResNet-101 achieves 42.8 AP on test-dev, outperforming almost all single-model detectors. The enhanced version reaches 50.9 AP. Compared to baseline, Cascade R-CNN shows 6.1 points AP80 and 8.7 points AP90 improvement. Ablation studies reveal: using u=0.5 for all stages still improves baseline, confirming that distribution changes justify stage-specific training; three stages provide best cost-performance tradeoff (38.9 AP). The method demonstrates consistent 2-4 point improvements across Faster R-CNN, R-FCN, FPN+, PASCAL VOC, KITTI, CityPersons, and WiderFace.

Cascade R-CNN proposes a multi-stage object detection framework addressing the rarely-explored high-quality detection problem. By sequentially training stages on progressively refined hypotheses with increasing IoU thresholds, the approach overcomes overfitting and quality mismatch challenges. The method demonstrates "very consistent performance gains on multiple challenging datasets" and "many object detectors, backbone networks, and techniques," suggesting broad utility for future detection and instance segmentation research.