We present a new method that views object detection as a direct set prediction problem. Our approach, called DEtection TRansformer (DETR), streamlines the detection pipeline, effectively removing the need for many hand-designed components like non-maximum suppression (NMS) or anchor generation that explicitly encode prior knowledge about the task. The main ingredients of DETR are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel.

The goal of object detection is to predict a set of bounding boxes and category labels for each object of interest. Modern detectors address this set prediction task in an indirect way, by defining surrogate regression and classification problems on a large set of proposals, anchors, or window centers. Their performance is significantly influenced by postprocessing steps to collapse near-duplicate predictions, by the design of the anchor sets, and by the heuristics that assign target boxes to anchors. To simplify these pipelines, we propose a direct set prediction approach to bypass the surrogate tasks.

Our DEtection TRansformer (DETR) makes predictions through two key ingredients: a set loss function that performs bipartite matching between predicted and ground-truth objects, eliminating the need for NMS, and a transformer-based architecture that models all pairwise interactions between elements using attention mechanisms. The self-attention mechanisms in the transformer allow DETR to perform global reasoning about the image, enabling it to capture long-range dependencies and remove duplicate detections without NMS.

DETR infers a fixed-size set of N predictions, in a single pass through the decoder, where N is set to be significantly larger than the typical number of objects in an image. One of the main difficulties is to score predicted objects with respect to the ground truth. Our loss produces an optimal bipartite matching between predicted and ground truth objects, and then optimizes object-specific losses. We search for a permutation of N elements with the lowest cost using the Hungarian algorithm. The matching cost takes into account both the class prediction and the similarity of predicted and ground truth boxes.

The overall DETR architecture is surprisingly simple and consists of three main components: a CNN backbone to extract a compact feature representation, an encoder-decoder transformer, and a simple feed forward network (FFN) that makes the final detection prediction. The encoder receives a flattened sequence of features supplemented with positional encodings and applies self-attention. The decoder receives N learned object queries and attends to the encoder output through cross-attention, producing N output embeddings that are independently decoded into box coordinates and class labels.

We evaluate DETR on the COCO 2017 detection dataset. Using a ResNet-50 backbone, DETR achieves 42.0 AP, which is competitive with the well-established Faster R-CNN baseline at 42.0 AP after training schedule optimizations. With a ResNet-101 backbone, DETR achieves 43.5 AP. Notably, DETR significantly outperforms Faster R-CNN on large objects (AP_L), gaining +7.8 AP, likely due to the global reasoning performed by the transformer's self-attention. However, DETR shows lower performance on small objects, achieving only 20.5 AP_S compared to Faster R-CNN's 22.5.

We further demonstrate the versatility of DETR by extending it to panoptic segmentation. By adding a simple segmentation head on top of the decoder outputs, DETR achieves competitive panoptic quality (PQ) results, demonstrating that the unified architecture can be naturally extended to related tasks. We also perform extensive ablation studies on the number of encoder/decoder layers, the role of positional encodings, and the importance of the FFN in the transformer, confirming that each component contributes to the final performance.

We have presented DETR, a new design for object detection systems based on transformers and bipartite matching loss for direct set prediction. DETR achieves comparable results to an optimized Faster R-CNN baseline on COCO while being conceptually simpler and requiring no hand-designed components. DETR is easily extensible to panoptic segmentation, demonstrating its potential as a unified architecture for visual recognition. We believe that the transformer-based approach opens up new possibilities for end-to-end object detection and beyond.