Feature pyramids are a basic component in recognition systems for detecting objects at different scales. But recent deep learning object detectors have avoided pyramid representations, in part because they are compute and memory intensive. In this paper, we exploit the inherent multi-scale, pyramidal hierarchy of deep convolutional networks to construct feature pyramids with marginal extra cost. A top-down architecture with lateral connections is developed for building high-level semantic feature maps at all scales. This architecture, called a Feature Pyramid Network (FPN), shows significant improvement as a generic feature extractor in several applications. Using FPN in a basic Faster R-CNN system, our method achieves state-of-the-art single-model results on the COCO detection benchmark without bells and whistles.

Recognizing objects at vastly different scales is a fundamental challenge in computer vision. Historically, featurized image pyramids (i.e., feature pyramids built upon image pyramids) were standard practice. These image pyramids have a key advantage: all levels of the pyramid are semantically strong, including the high-resolution levels. But featurized image pyramids have a critical limitation: inference time increases considerably (e.g., by four times), making this approach impractical for real applications. Moreover, training deep networks end-to-end on an image pyramid is infeasible in terms of memory. Deep ConvNets compute a feature hierarchy layer by layer with inherent pyramidal structure. However, this introduces large semantic gaps caused by different depths — high-resolution maps have low-level features that reduce representational capacity for object detection.

The Single Shot Detector (SSD) attempted leveraging the ConvNet hierarchy as a pyramid representation but foregoes reusing already computed layers and instead builds the pyramid starting from high up in the network, missing opportunities for detecting small objects. Our goal is to naturally leverage the pyramidal shape of a ConvNet's feature hierarchy while creating a feature pyramid that has strong semantics at all scales. To achieve this, we develop an architecture that combines low-resolution, semantically strong features with high-resolution, semantically weak features via a top-down pathway and lateral connections. The result is an in-network feature pyramid that can replace featurized image pyramids without sacrificing representational power, speed, or memory.

Hand-engineered features such as SIFT and HOG were computed densely over entire image pyramids in the era of traditional object detection. Modern deep ConvNet detectors like Fast R-CNN and Faster R-CNN advocate single-scale features as an accuracy-speed tradeoff. Recent work employs lateral/skip connections including U-Net, SharpMask, and Stacked Hourglass networks. However, these methods differ fundamentally from FPN: they produce a single high-level feature map rather than independently making predictions at all pyramid levels. Methods like SSD and MS-CNN predict objects at multiple layers of the feature hierarchy but without combining features or scores across levels.

The bottom-up pathway is the feedforward computation of the backbone ConvNet, which computes a feature hierarchy consisting of feature maps at several scales with a scaling step of 2. There are often many layers producing output maps of the same size and we say these layers are in the same network stage. For our feature pyramid, we define one pyramid level for each stage. We choose the output of the last layer of each stage as our reference set of feature maps to create the pyramid. For ResNets, we use the feature activations output by each stage's last residual block, denoting the outputs of {C_2, C_3, C_4, C_5} with strides of {4, 8, 16, 32} pixels relative to the input image.

The top-down pathway hallucinates higher resolution features by upsampling spatially coarser, but semantically stronger, feature maps from higher pyramid levels. These features are then enhanced with features from the bottom-up pathway via lateral connections. Each lateral connection merges feature maps of the same spatial size from the bottom-up pathway and the top-down pathway. The bottom-up feature map undergoes a 1x1 convolution to reduce channel dimensions, and the top-down map is upsampled by 2x using nearest neighbor interpolation. These two maps are then merged by element-wise addition. A 3x3 convolution is appended to each merged map to reduce the aliasing effect of upsampling, producing the final feature maps {P_2, P_3, P_4, P_5}. All pyramid levels share the same channel dimension d = 256.

We emphasize that our design is deliberately kept simple, and we have found that our model is robust to many design choices. We have experimented with more sophisticated blocks (e.g., using multi-layer perceptrons in lateral connections) and achieved marginally better results, but the added complexity is not worth the cost. The shared classifiers and regressors operate at all pyramid levels with the same parameters, similar to traditional featurized image pyramids. This weight sharing also works because all levels of the pyramid use shared semantics from the enriched feature maps. The simplicity of FPN makes it a generic component that can be readily plugged into various detection and segmentation frameworks.

RPN experiments show FPN improved Average Recall (AR_1k) to 56.3, an 8.0 point increase over single-scale RPN baseline. Small object AR improved by 12.9 points. Ablation studies confirmed that removing top-down pathways severely degrades performance; removing lateral connections reduces AR_1k by 10 points. For object detection, FPN improved AP by 2.3 points and AP@0.5 by 3.8 points over single-scale Faster R-CNN baselines on COCO minival. The method surpassed 2016 COCO challenge winners without image pyramids or techniques like iterative regression or hard negative mining. For segmentation proposals, FPN achieved 48.1 AR for mask generation, outperforming previous methods by over 8.3 points while running at 6-7 FPS.

We have presented a clean and simple framework for building feature pyramids inside ConvNets. Our method shows significant improvements over several strong baselines and competition winners. Thus, it provides a practical solution for research and applications of feature pyramids, without the need of computing image pyramids. We have demonstrated that despite the strong representational power of deep ConvNets and their implicit robustness to scale variation, it is still critical to explicitly address multi-scale problems using pyramid representations.