This paper presents Online Hard Example Mining (OHEM), a training algorithm for region-based ConvNet object detectors. The authors observe that "detection datasets contain an overwhelming number of easy examples and a small number of hard examples". Their method automatically selects difficult training instances by computing losses for all proposals and selecting the hardest ones for backpropagation. OHEM eliminates several manual training heuristics and yields improvements to 78.9% mAP on PASCAL VOC 2007 and 76.3% on VOC 2012 when combined with complementary techniques.

Region-based ConvNet detectors such as Fast R-CNN have become the dominant paradigm for object detection. Training these detectors requires sampling Region of Interest (RoI) proposals, but the vast majority of proposals are easy background examples that contribute little to learning. Current practice relies on ad-hoc heuristics: a fixed foreground-to-background ratio of 1:3, a background IoU lower threshold (bg_lo = 0.1) to avoid easy negatives, and random sampling within these constraints. These heuristics are "suboptimal because they ignore valuable hard examples".

Hard negative mining has a long history in object detection, dating back to bootstrapping methods for training SVMs and DPMs. Classical approaches use an alternating strategy: train the model, apply it to find hard negatives, add these to the training set, and retrain. This "freeze-and-mine" procedure is expensive and incompatible with modern SGD-based training of deep networks. Recent works like Fast R-CNN avoid explicit hard mining by using random sampling with ratio constraints, which is simpler but suboptimal. OHEM bridges this gap by performing hard example selection within the standard SGD framework without alternating phases.

OHEM modifies standard SGD by sampling training examples according to a non-uniform distribution based on current loss values. For each mini-batch, all RoI proposals from the input images undergo a forward pass to compute per-proposal losses. The algorithm then selects the B/N hardest examples by sorting losses in descending order. To avoid redundant overlapping regions, non-maximum suppression (NMS) with an IoU threshold of 0.7 is applied before selection. Only the selected hard examples participate in the backward pass. This approach "freezes the model for only one mini-batch" — maintaining the normal SGD update frequency while ensuring every gradient update is informed by the most challenging examples.

The authors propose a dual-network architecture for efficient implementation. A read-only RoI network performs forward passes on all proposals to compute losses, while a standard network processes only the selected hard examples for gradient computation. The read-only network shares convolutional weights with the training network but does not accumulate gradients, balancing memory efficiency with training speed. This eliminates the need to allocate memory for all proposal gradients, making the approach scalable to images with thousands of proposals.

OHEM is evaluated on PASCAL VOC and MS COCO. On PASCAL VOC, OHEM improves Fast R-CNN baseline from 67.2% to 69.9% mAP on VOC 2007 (2.7 points) and from 65.7% to 69.8% on VOC 2012 (4.1 points). On the more challenging MS COCO dataset, the standard metric improves from 19.7% to 22.6% AP, with a notable 4.9-point boost for medium-sized objects. Importantly, OHEM eliminates three training heuristics without loss: it removes the background IoU lower threshold (bg_lo), eliminates the fixed 1:3 foreground-to-background ratio, and automatically balances difficult examples across classes.

OHEM demonstrates that automatic hard example selection during training is a simple yet effective strategy for improving region-based object detectors. By leveraging the loss computed during the forward pass to select informative examples, the method eliminates several ad-hoc training heuristics while consistently improving detection accuracy. The approach is general-purpose and can be applied to any region-based detector, making it a practical contribution to the object detection training pipeline.