The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage methods to date. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design a simple dense detector called RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors.

Current state-of-the-art object detectors are based on a two-stage, proposal-driven mechanism. As popularized in the R-CNN framework, the first stage generates a sparse set of candidate proposals, and the second stage classifies each proposal. Through a sequence of advances, this two-stage framework consistently achieves top accuracy on the challenging COCO benchmark. Despite the success of two-stage methods, a natural question is: could a simple one-stage detector achieve similar accuracy? One-stage detectors are applied over a regular, dense grid of possible object locations, sizes, and aspect ratios. Recent work on one-stage methods, such as YOLO and SSD, demonstrates promising results, yielding faster detectors with accuracy within 10-40% relative to state-of-the-art two-stage methods.

We push the envelope further in this paper: we show for the first time that a one-stage object detector can match or surpass two-stage detectors in accuracy. We identify class imbalance during training as the main obstacle impeding one-stage methods from achieving state-of-the-art accuracy. Class imbalance is addressed in two-stage detectors by a two-stage cascade (proposals reduce candidates from ~100k to ~1-2k) and by biased minibatch sampling (e.g., 1:3 foreground-to-background ratio). In one-stage detectors, similar heuristics are applied but they are inefficient as the training procedure is still dominated by easily classified background examples.

Two-stage detectors such as Faster R-CNN use a Region Proposal Network (RPN) followed by a classification network. The two-stage approach addresses class imbalance by generating ~1-2k proposals that filter out most background, and by using fixed foreground-to-background sampling ratios. One-stage detectors like SSD and YOLO predict detections directly from a dense grid. These methods address the imbalance issue through hard negative mining or bootstrapping, which select hard examples for training. Online Hard Example Mining (OHEM) is a representative approach that selects the highest-loss examples for each mini-batch, but still relies on heuristic sampling strategies. In contrast, our focal loss addresses class imbalance directly through the loss function without any sampling or mining.

We start from the cross entropy (CE) loss for binary classification: CE(p, y) = -log(p_t), where p_t = p if y = 1, else 1-p. A notable property of CE is that even easily-classified examples (p_t >> 0.5) incur a non-trivial loss. When summed over a large number of easy examples, these small loss values can overwhelm the rare class. We propose to add a modulating factor (1 - p_t)^gamma to the cross entropy loss, with tunable focusing parameter gamma >= 0: FL(p_t) = -(1 - p_t)^gamma * log(p_t). The focal loss has two key properties: (1) when an example is misclassified and p_t is small, the modulating factor is near 1 and the loss is unaffected; (2) as p_t approaches 1, the factor goes to 0 and the loss for well-classified examples is down-weighted. For example, with gamma = 2, an example classified with p_t = 0.9 has 100x lower loss compared to CE. In practice, we use an alpha-balanced variant: FL(p_t) = -alpha_t * (1 - p_t)^gamma * log(p_t).

An important practical consideration is model initialization. For models trained with CE, random initialization typically results in equal probability for all classes. Under this initialization, the loss due to the frequent class (background) can dominate total loss and cause instability in early training. To counter this, we introduce a "prior" concept for the value of p estimated by the model for the rare class (foreground) at the start of training. We set this prior to pi = 0.01, which ensures that the loss from the dominant class is low at initialization, improving training stability. We find that this initialization is critical for stable training with both CE and FL, improving AP by ~0.7 points.

RetinaNet is a single, unified network composed of a backbone network and two task-specific subnetworks. The backbone is a Feature Pyramid Network (FPN) built on top of ResNet, which generates a rich, multi-scale feature pyramid from levels P3 through P7. The first subnet is the classification subnet: a small FCN attached to each FPN level, consisting of four 3x3 conv layers with 256 channels (each followed by ReLU), terminated by a 3x3 conv with KA outputs (K classes, A anchors) and sigmoid activation. The second subnet is the box regression subnet: an identical structure parallel to the classification subnet, but outputting 4A values per spatial location. Anchors comprise 9 per level (3 aspect ratios x 3 scales), covering a scale range of 32 to 813 pixels. Importantly, our simple detector achieves top results not based on innovations in network design but due to our novel loss.

All experiments are conducted on the COCO benchmark. In ablation studies with ResNet-50, standard CE with proper initialization achieves 30.2 AP. Adding alpha-balanced CE improves this to 31.1 AP (best alpha = 0.75). Focal loss with gamma = 2 and alpha = 0.25 achieves 34.0 AP, a 2.9 point improvement over alpha-balanced CE. FL outperforms the best variant of online hard example mining (OHEM) by over 3 points AP (36.0 vs. 32.8 AP with ResNet-101). For state-of-the-art comparison, RetinaNet-101-800 achieves 39.1 AP on COCO test-dev. With a ResNeXt-101-FPN backbone, it reaches 40.8 AP, surpassing all prior one-stage and two-stage detectors including Faster R-CNN with FPN. In terms of speed, RetinaNet-101-600 achieves 36.2 AP at 122ms per image, while Faster R-CNN+FPN achieves the same AP at 172ms — RetinaNet is 29% faster at comparable accuracy.

In this paper, we identified class imbalance as the primary cause for the accuracy gap between one-stage and two-stage object detectors. To address this, we proposed the focal loss which applies a modulating term to the cross entropy loss to focus learning on hard misclassified examples. Our approach is simple and highly effective. We demonstrated its effectiveness by designing RetinaNet, a simple one-stage object detector that achieves state-of-the-art accuracy, surpassing all previously published two-stage detectors. We hope the simplicity and effectiveness of focal loss will benefit other domains with severe class imbalance.