This paper presents UP-DETR, the first unsupervised pre-training approach specifically designed for transformer-based object detection. The core idea is a novel pretext task called random query patch detection, where the model learns to detect randomly cropped patches from the input image using the transformer decoder's object queries. This pre-training enables DETR-like models to converge faster and perform significantly better, especially in low-data regimes. Experiments show 6.7% AP improvement on PASCAL VOC with only 10% labeled data.

DETR (DEtection TRansformer) reformulated object detection as a direct set prediction problem, eliminating the need for anchor boxes, non-maximum suppression, and hand-crafted components. However, DETR suffers from slow convergence — requiring 500 epochs to train, approximately 10-20x more than Faster R-CNN. Furthermore, DETR's performance degrades significantly when labeled training data is limited, a common scenario in many practical applications.

While unsupervised pre-training has shown remarkable success in NLP (BERT, GPT) and image classification (MoCo, SimCLR), no prior work has addressed unsupervised pre-training specifically for the transformer decoder in object detection. The challenge lies in designing a pretext task that aligns with the unique structure of DETR's decoder — particularly its learnable object queries that must localize and classify objects simultaneously.

Self-supervised learning methods like MoCo and SimCLR learn visual representations through contrastive learning on image-level features. While effective for classification backbones, these methods pre-train only the CNN encoder and do not address the transformer decoder's object queries. DETR and its variants (Deformable DETR) have improved detection performance but still rely on supervised ImageNet pre-training for the backbone and random initialization for the transformer components.

The pretext task is random query patch detection: given an input image, the method randomly crops a patch and uses it as a query, asking the model to predict the location (bounding box) of this patch within the original image. The cropped patch is encoded by the frozen CNN backbone and used to initialize the object query in the transformer decoder. This task naturally teaches the decoder to associate visual features with spatial locations — the exact capability needed for object detection.

To better utilize DETR's multiple object queries, the authors extend the pretext task to multi-query localization, where multiple patches are randomly cropped and each is assigned to a different object query for simultaneous detection. An attention mask is applied to prevent cross-attention between different query patches, ensuring each query independently learns to localize its assigned patch. Additionally, feature reconstruction is added as an auxiliary task, where the decoder must reconstruct the frozen CNN features of the cropped patches, encouraging the model to learn richer semantic representations beyond mere localization.

On PASCAL VOC with only 10% labeled data, UP-DETR improves DETR by 6.7% AP. On COCO object detection, UP-DETR achieves 43.1% AP with ResNet-50 backbone, surpassing vanilla DETR (42.0% AP) by 1.1%. The pre-training also accelerates convergence: UP-DETR reaches DETR's 150-epoch performance in just 60 epochs. Ablation studies demonstrate that multi-query localization contributes +0.6% AP and feature reconstruction adds another +0.4% AP over single-query pre-training. On the full COCO dataset, the improvement is more modest (1.1% AP), suggesting that pre-training benefits are most pronounced in data-scarce scenarios.

UP-DETR demonstrates that unsupervised pre-training can be effectively designed for transformer-based object detectors through the random query patch detection pretext task. The approach provides significant improvements in data-scarce settings and accelerates the notoriously slow convergence of DETR. By aligning the pre-training task with the downstream detection task's query-based architecture, UP-DETR bridges the gap between self-supervised representation learning and end-to-end object detection.