We propose FCOS, a fully convolutional one-stage object detector that solves object detection in a per-pixel prediction fashion, anchor-free and proposal-free, analogous to semantic segmentation. By eliminating predefined anchor boxes, FCOS avoids the computation related to anchor boxes such as the commonly needed IoU computation and matching during training, and also avoids all hyper-parameters related to anchor boxes. With the only post-processing being non-maximum suppression (NMS), FCOS with ResNeXt-64x4d-101 achieves 44.7% AP on the MS-COCO benchmark, surpassing previous one-stage detectors with the advantage of being much simpler.

Current mainstream detectors rely on anchor boxes, which introduce several drawbacks: detection performance is sensitive to anchor sizes and aspect ratios, affecting performance by ~4% AP on COCO; fixed-scale anchor boxes struggle with shape variations, particularly small objects; dense anchor box placement creates excessive negative samples, exacerbating class imbalance; and anchor-based methods involve complicated IoU computation. The authors ask: can object detection adopt the same per-pixel prediction framework that has succeeded in semantic segmentation, depth estimation, keypoint detection, and counting?

Previous FCN-based detectors like DenseBox required image resizing to fixed scales and proved unsuitable for generic detection due to overlapping bounding box ambiguity. FCOS addresses this through multi-level FPN prediction, largely eliminating the ambiguity. A novel "center-ness" branch predicts the deviation of each location from its corresponding object center, suppressing low-quality detections. This simple yet effective component enables the FCN detector to outperform anchor-based alternatives under identical settings.

Anchor-based Detectors inherit from traditional sliding-window approaches. Faster R-CNN popularized anchor boxes, followed by SSD and YOLOv2. However, anchor-based methods require extensive hyperparameter tuning regarding anchor shapes and positive/negative sample labeling via IoU thresholds. Anchor-free Detectors: YOLOv1 predicted boxes at object centers but suffered low recall, prompting YOLOv2 to adopt anchors. CornerNet detects corner pairs but requires complicated grouping procedures. FCOS leverages all points within ground-truth boxes as positive samples, using center-ness to suppress low-quality detections.

Unlike anchor-based methods, FCOS directly regresses target bounding boxes at each spatial location, treating them as training samples similar to semantic segmentation. A location is positive if it falls within any ground-truth box, negative otherwise. For each positive location, the network predicts a 4D vector t* = (l*, t*, r*, b*) representing distances from the location to the four sides of the bounding box. The training loss combines focal loss for classification and IoU loss for bounding box regression. FCOS outputs 9x fewer parameters per location than anchor-based detectors using 9 anchors per location.

Multi-level prediction using five FPN levels {P3, P4, P5, P6, P7} with strides 8, 16, 32, 64, and 128 addresses two concerns: low best possible recall (BPR) from large feature map strides, and ambiguity from overlapping ground-truth boxes. Object sizes are assigned to appropriate levels by constraining the regression range — if max(l*, t*, r*, b*) exceeds a level-specific threshold, the location becomes a negative sample. With FPN, FCOS achieves 98.40% BPR, nearly matching RetinaNet's best (99.23%), and ambiguous samples are reduced from 23.16% to 7.14%.

Despite multi-level prediction, low-quality predicted boxes emerge from locations far from object centers. A parallel single-layer branch predicts "center-ness", measuring the normalized distance from each location to its corresponding object center: centerness* = sqrt[(min(l*,r*)/max(l*,r*)) * (min(t*,b*)/max(t*,b*))]. Center-ness ranges from 0 to 1, trained with binary cross-entropy loss. During testing, predicted center-ness is multiplied with the classification score, down-weighting far-from-center predictions. This improves AP from 33.5% to 37.1%, enabling FCOS to surpass RetinaNet (35.9%).

On MS-COCO test-dev, FCOS achieves remarkable results across multiple backbones: ResNet-101-FPN: 41.5% AP versus RetinaNet 39.1% AP (+2.4%); ResNeXt-64x4d-101-FPN: 43.2% AP, surpassing CornerNet (40.5%); and with additional improvements, FCOS reaches 44.7% AP. As a Region Proposal Network (RPN) replacement, FCOS improves AR_100 by 8.1% and AR_1k by 3.4% over anchor-based RPNs. The center-ness branch alone accounts for +3.6% AP improvement, and a separate branch outperforms computing center-ness from predicted regression vectors.

FCOS presents a simpler, anchor-free, proposal-free alternative to mainstream anchor-based detectors, achieving state-of-the-art one-stage performance with significantly fewer design complexities. By eliminating anchor computation and hyperparameters, the method unifies detection with dense prediction tasks. The framework extends readily to instance-level tasks including instance segmentation and keypoint detection, serving as a strong baseline for instance-wise prediction problems. FCOS demonstrates that anchor boxes, long considered essential for competitive detection, are in fact unnecessary.