In this paper we present Mask DINO, a unified object detection and segmentation framework. Mask DINO extends DINO (DETR with Improved Denoising Anchor Boxes) by adding a mask prediction branch which supports all image segmentation tasks (instance, panoptic, and semantic). It makes use of the query embeddings from DINO to dot-product with high-resolution pixel embedding maps for mask prediction. Building upon DINO, the key components include a unified query selection for detection and segmentation, a unified denoising training for both tasks, and a hybrid bipartite matching that enhances mutual benefits between detection and segmentation.

Mask DINO demonstrates significant improvements over all existing specialized and unified models. It achieves the best results among models with less than one billion parameters on all three segmentation tasks — with 54.5 AP on COCO instance segmentation, 59.4 PQ on COCO panoptic segmentation, and 60.8 mIoU on ADE20K semantic segmentation. Notably, with a ResNet-50 backbone, Mask DINO achieves 46.3 AP on COCO instance segmentation, outperforming Mask2Former by 2.6 AP, and 51.7 AP on COCO object detection, outperforming DINO itself by 0.8 AP, demonstrating the mutual benefits of detection and segmentation in a unified framework.

Object detection and image segmentation are fundamental computer vision tasks with different focuses. Detection aims to localize objects and predict bounding boxes and categories, while segmentation performs pixel-level grouping of different semantics, encompassing instance, panoptic, and semantic variants. Classical convolution-based algorithms achieved remarkable progress through specialized architectures — Faster RCNN for detection, Mask RCNN for instance segmentation, and FCN for semantic segmentation — but these lacked generalization across tasks.

Recently, Transformer-based DETR-like models have achieved significant progress on both detection and segmentation. DINO reached state-of-the-art detection results on COCO, while MaskFormer and Mask2Former unified all three segmentation tasks under a single architecture. However, in Transformer-based models, the best-performing detection and segmentation models are still not unified, which prevents task and data cooperation between detection and segmentation tasks. Simply adding DETR's segmentation head to DINO results in inferior instance segmentation results, and naive multi-task training actually hurts performance.

The authors pose two key questions: (1) Why can detection and segmentation not cooperate well in existing Transformer-based models? (2) How can we design a unified framework where the two tasks mutually benefit each other? To address these, they propose Mask DINO, which extends DINO with three key components: unified query selection that initializes queries using both box and mask predictions from the encoder, unified denoising training that extends denoising to segmentation masks, and hybrid bipartite matching that jointly considers box and mask losses. These designs ensure that detection and segmentation align from query initialization through training and matching, achieving genuine mutual benefits.

Transformer-based detectors have dominated recent progress in object detection. DETR pioneered the end-to-end set prediction approach, eliminating hand-crafted modules like non-maximum suppression (NMS) and anchor generation. DAB-DETR reformulated queries as 4D anchor boxes with layer-by-layer refinement. DN-DETR introduced denoising training to accelerate convergence. DINO combined improvements from both, achieving state-of-the-art results on COCO detection with 63.3 AP using a SwinL backbone. These advances in detection architecture provide a strong foundation that segmentation could potentially benefit from.

Image segmentation encompasses three main variants: instance segmentation (predicting a mask plus category per object), semantic segmentation (per-pixel classification including background), and panoptic segmentation (unifying both). Specialized architectures dominated historically: Mask RCNN and HTC for instance segmentation, FCN and U-Net for semantic, and separate panoptic models. Recent unified approaches, especially K-Net, MaskFormer, and Mask2Former, achieved remarkable multi-task performance using query-based Transformer architectures with mask classification, demonstrating that a single segmentation architecture can handle all three tasks.

Previous attempts at unified detection and segmentation include CNN-based models like Mask RCNN, HTC, and Panoptic FPN, which combined detection and segmentation branches. In the Transformer era, the original DETR also explored adding a segmentation head. However, simply adding a segmentation head to a state-of-the-art detector like DINO leads to performance degradation rather than improvement. The detection and segmentation tasks interfere with each other when naively combined, suggesting that careful architectural alignment is needed to achieve mutual benefits rather than mutual harm.

Mask DINO extends DINO by adding a parallel mask prediction branch alongside the existing box and label prediction branches. The framework maintains DINO's overall architecture: a backbone network extracts multi-scale features, a deformable Transformer encoder processes these features for feature enhancement, and a Transformer decoder takes content queries and positional queries to produce detection and segmentation outputs. The key architectural addition is a segmentation branch that constructs a high-resolution pixel embedding map by fusing backbone features at 1/4 resolution with upsampled encoder features at 1/8 resolution. Binary masks are produced by dot-producting content query embeddings with this combined pixel embedding map.

Formally, the mask prediction follows a mask classification paradigm. Let q_c denote the content query embedding from the decoder. The pixel embedding map M is constructed as: M = T(C_b) + F(C_e), where C_b represents the 1/4-resolution backbone feature, C_e represents the 1/8-resolution encoder feature, T is a lateral connection, and F is an upsampling operator. The binary mask m is then obtained by: m = q_c ⊗ M, where ⊗ denotes the dot product operation. This formulation is lightweight yet effective, adding minimal computational overhead to the existing detection pipeline.

A critical design in Mask DINO is the unified query selection mechanism. The method adopts three prediction heads (classification, detection, and segmentation) at the encoder output level, extending DINO's original two-head (classification and detection) query selection. The top-ranked encoder features, scored by classification confidence, are selected to initialize the decoder content queries. This provides the decoder with strong initial priors from the encoder's dense predictions, significantly accelerating convergence and improving performance. In ablation studies, this unified query selection enables 39.6 AP mask predictions even at decoder layer 0, compared to only 1.1 AP achieved by Mask2Former at the same stage.

A novel insight is that predicted masks from the encoder are initially more accurate than predicted boxes, especially for irregularly shaped objects. Mask DINO introduces mask-enhanced anchor box initialization: the minimum bounding rectangle of the predicted mask is used to derive an enhanced anchor box, replacing the directly predicted box for decoder initialization. This creates a virtuous cycle where mask predictions improve box initialization, and better box anchors in turn benefit subsequent mask refinement. Ablation results show that mask-enhanced initialization boosts layer-0 mask AP from 25.6 to 41.2 (+15.6) and improves final detection AP by 1.2 points.

DN-DETR demonstrated that denoising training — feeding noised ground-truth boxes and training the model to reconstruct the original — significantly accelerates DETR-like model convergence. Mask DINO extends this idea to segmentation: ground-truth boxes are treated as noised versions of masks, and the model is trained to reconstruct the original mask given a noised box as the positional query. The denoising process includes label flipping with probability 0.2, center shifting constrained to half the box dimensions scaled by lambda_1=0.4, and box scaling within [(1-lambda_2), (1+lambda_2)] where lambda_2=0.4. This unified denoising bridges the gap between detection and segmentation during training, forcing the model to learn the correspondence between boxes and masks.

The effectiveness of unified denoising is demonstrated through convergence speed improvements. In standard training without denoising, the model requires many more epochs to achieve comparable performance. With unified denoising, Mask DINO achieves 44.2 AP on instance segmentation in only 24 epochs, which is 0.5 AP higher than Mask2Former's result of 43.7 AP at 50 epochs. This demonstrates that denoising training not only accelerates convergence but also provides a regularization effect that improves final performance. The denoising queries and matching queries are processed in separate groups to prevent information leakage between them.

Standard bipartite matching in DETR-like detectors uses only classification and box regression losses to establish the one-to-one correspondence between predictions and ground truths. However, when both box and mask predictions exist, using only box loss for matching may lead to inconsistent box-mask pairs — a prediction might have a well-matched box but a poorly aligned mask. Mask DINO introduces hybrid bipartite matching that incorporates mask prediction loss alongside classification and box losses in the matching cost: L = lambda_cls * L_cls + lambda_box * L_box + lambda_mask * L_mask. This ensures that the matching considers both detection and segmentation quality simultaneously, producing more consistent predictions.

Ablation studies validate the effectiveness of hybrid matching. With box-only matching, detection achieves 44.4 AP but mask quality is suboptimal. With mask-only matching, detection drops significantly to 40.2 AP as the matching ignores box localization quality. The hybrid matching achieves 44.5 AP detection and 41.4 AP masks, demonstrating that considering both modalities in matching is essential for maintaining strong performance on both tasks simultaneously. Furthermore, decoupled box prediction is employed for panoptic segmentation, where box loss and matching are removed for "stuff" categories (amorphous regions like sky and road) while box predictions are retained for deformable attention feature extraction.

On COCO instance segmentation and object detection, Mask DINO with a ResNet-50 backbone achieves 46.3 AP on instance segmentation at 50 epochs, surpassing Mask2Former's 43.7 AP by 2.6 points. Simultaneously, it achieves 51.7 AP on object detection, outperforming DINO's 50.9 AP by 0.8 points. This result is particularly significant because it demonstrates that adding segmentation capability to a top detection model does not degrade but actually improves detection performance. Even at 24 epochs, Mask DINO achieves 44.2 AP on masks, already surpassing Mask2Former's 50-epoch result. With a SwinL backbone, the gaps further widen: 52.3 AP masks (+2.2 over Mask2Former) and 59.0 AP detection (+0.5 over DINO).

On COCO panoptic segmentation, Mask DINO achieves 53.0 PQ at 50 epochs with ResNet-50, exceeding Mask2Former's 51.9 PQ by 1.1 points. At the 12-epoch setting, the improvement is even more pronounced: 49.0 PQ versus Mask2Former's 46.9 PQ (+2.1), indicating faster convergence benefits from detection-segmentation cooperation. The decoupled box prediction for "stuff" categories proves essential for panoptic performance, as enforcing box constraints on amorphous regions like sky or grass would introduce misleading supervision signals. With a SwinL backbone, Mask DINO reaches 58.3 PQ, improving upon Mask2Former by 0.5 points.

On ADE20K semantic segmentation, Mask DINO with ResNet-50 achieves 48.7 mIoU, outperforming Mask2Former's 47.2 mIoU by 1.5 points. On Cityscapes, it reaches 80.0 mIoU versus Mask2Former's 79.4 mIoU (+0.6). In the large-scale comparison, using a SwinL backbone with Objects365 detection pre-training, Mask DINO establishes new state-of-the-art results among sub-billion parameter models across all three tasks: 54.5 AP on COCO instance segmentation, 59.4 PQ on COCO panoptic segmentation, and 60.8 mIoU on ADE20K semantic segmentation. The ability to leverage large-scale detection pre-training for segmentation is a unique advantage that previous specialized segmentation models could not exploit.

Comprehensive ablation studies validate each component's contribution. Removing all three proposed components (unified query selection, unified denoising, and hybrid matching) reduces detection AP by 4.0 points and mask AP by 2.7 points. Individually, query selection provides the most substantial gains: the encoder-based initialization enables 39.6 AP mask predictions at decoder layer 0, compared to merely 1.1 AP from Mask2Former's random initialization. By decoder layer 3, Mask DINO already achieves 44.0 AP, which is competitive with Mask2Former's final performance. The multi-scale feature design also proves important: single-scale features yield 45.8 AP boxes and 45.1 AP masks, while four-scale features improve to 50.5 AP boxes and 46.0 AP masks.

A key ablation examines task cooperation between detection and segmentation. Detection-only training yields 50.1 AP on boxes; segmentation-only training yields 43.3 AP on masks. When jointly trained, detection improves to 50.5 AP (+0.4) and segmentation improves to a much larger extent, confirming the asymmetric but genuine mutual benefit between the two tasks. The detection task provides strong localization priors that benefit segmentation, while segmentation provides fine-grained spatial information that moderately improves detection. This experiment directly answers the paper's motivating question: with proper architectural alignment, detection and segmentation not only avoid interference but actively cooperate.

Further analysis across decoder layers reveals the progressive refinement behavior. With unified query selection, mask AP improves from 39.6 at layer 0 to 44.0 at layer 3 and reaches peak performance at layer 5. Similarly, detection AP progresses from moderate to strong across layers, confirming that both tasks benefit from the iterative refinement mechanism of the Transformer decoder. The multi-decoder layer design in Mask DINO proves especially important for aligning box and mask predictions: early layers can leverage mask-enhanced initialization for better box anchors, while later layers fine-tune both predictions jointly.

Mask DINO presents a unified Transformer-based framework that demonstrates detection and segmentation can achieve mutual benefit through careful architectural design. By extending DINO with a mask prediction branch, unified query selection, unified denoising training, and hybrid bipartite matching, the framework achieves state-of-the-art results across all three segmentation tasks (instance, panoptic, and semantic) among sub-billion parameter models. The results demonstrate that a unified model can outperform specialized models on their respective tasks, challenging the conventional wisdom that task-specific architectures are necessary for peak performance.

A particularly significant implication is that the unified framework enables segmentation models to benefit from large-scale detection pre-training datasets such as Objects365, a capability that previous specialized segmentation architectures could not leverage. This opens new avenues for scaling segmentation performance through detection data, which is typically more abundant and cheaper to annotate than pixel-level segmentation labels. The framework's design philosophy — aligning task representations from initialization through training to matching — provides a template for unifying other complementary vision tasks in the future.