Image segmentation is about grouping pixels with different semantics, e.g., category or instance membership. Each choice of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance, or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions, multi-scale high-resolution features to handle small objects, and optimization improvements such as switching the order of self- and cross-attention, making query features learnable, and removing dropout. Without bells and whistles, Mask2Former sets a new state-of-the-art on all four popular datasets for every segmentation task. Most notably it achieves 57.8 PQ on COCO panoptic segmentation, 50.1 AP on COCO instance segmentation, and 57.7 mIoU on ADE20K semantic segmentation.

Image segmentation groups pixels with different semantics. Depending on the specific semantics, the community has defined different segmentation tasks: semantic segmentation assigns a class label to each pixel; instance segmentation detects and segments each object instance; and panoptic segmentation unifies both by assigning every pixel a semantic label and grouping pixels into object instances. Despite the fact that these tasks only differ in the definition of output semantics, current research develops specialized architectures and loss functions for each task, fragmenting the segmentation community and tripling the research effort.

Recent work on universal image segmentation architectures such as DETR and MaskFormer has shown that mask classification — predicting a set of binary masks each with an associated class label — can address all three tasks. However, these universal architectures still lag behind the best specialized models, especially on instance segmentation. For example, MaskFormer is more than 9 AP behind the leading instance segmentation approach (Swin-HTC++) while requiring over 4 times more training epochs (300 vs. 72). Furthermore, MaskFormer's training cannot fit multiple images on a 32GB GPU, limiting accessibility. These practical gaps motivate us to ask: what makes a good universal architecture for image segmentation?

We present Mask2Former, which improves upon MaskFormer with three key modifications to the Transformer decoder. First, we propose masked attention to replace the standard cross-attention in the Transformer decoder. Unlike standard cross-attention which attends to the full feature map, masked attention restricts attention to localized features centered around predicted segments, which we find converges 6 times faster and improves performance. Second, we utilize multi-scale high-resolution features in the Transformer decoder via an efficient round-robin strategy that feeds one scale to each layer in succession. Third, we propose optimization improvements including switching the order of self- and cross-attention, making query features learnable, and removing dropout. Together with a point-based loss calculation that reduces training memory by 3 times, these changes yield a model that sets new state-of-the-art results on panoptic, instance, and semantic segmentation across four datasets.

Per-pixel classification has been the dominant paradigm for semantic segmentation since FCN, where a classification loss is applied independently to each output pixel. Subsequent methods improve upon FCN by incorporating contextual modules, dilated convolutions, and self-attention layers. In contrast, mask classification — used in Mask R-CNN and its descendants — predicts a set of binary masks, each associated with a single class prediction. While per-pixel classification dominates semantic segmentation and mask classification dominates instance segmentation, MaskFormer demonstrated that mask classification is sufficiently general to address semantic segmentation as well, unifying the two paradigms under one framework.

The pursuit of universal image segmentation has been accelerated by Transformer-based architectures. DETR introduced end-to-end set prediction for object detection using a Transformer encoder-decoder with bipartite matching loss. Subsequent works including K-Net use dynamic kernels for unified segmentation. However, a persistent concern is that Transformers converge slowly: standard DETR requires 500 training epochs. Analysis by prior work suggests this slow convergence stems from cross-attention attending globally to the entire image feature map, where most regions are irrelevant to each query. Deformable DETR addresses this via deformable attention on sparse sampling points, while our masked attention offers a complementary approach by using predicted masks to constrain the attention region.

Panoptic segmentation methods have evolved from combining separate semantic and instance segmentation networks with heuristic merging to more unified approaches. Panoptic-FPN adds a semantic segmentation branch to Mask R-CNN, while Panoptic-DeepLab builds upon DeepLab with an instance center prediction head. More recent approaches like MaX-DeepLab use dual-path transformers for end-to-end panoptic segmentation. Despite these advances, panoptic methods remain architecturally distinct from pure instance or semantic methods, and performance gaps persist when universal architectures are applied to individual tasks. Mask2Former aims to close these gaps by improving the Transformer decoder design rather than introducing entirely new architectures.

The Transformer decoder in MaskFormer uses standard cross-attention where each query attends to all spatial locations in the image feature map. Formally, given query features Q, image features as key K and value V, the standard cross-attention computes softmax(Q K^T) V over all positions. This global attention pattern forces each query to aggregate information from the entire feature map, including large irrelevant regions. Prior analysis has shown that this is a major cause of slow convergence in DETR-like models, as queries must gradually learn to attend to relevant regions from scratch. Our key observation is that local features, guided by predicted mask regions, are sufficient for segmentation, and global context can emerge through self-attention among queries.

We propose masked attention, which modifies the standard cross-attention by introducing an attention mask derived from the predicted mask of the previous Transformer decoder layer. At each feature location (x, y), the attention mask is set to 0 if the location falls within the predicted foreground region (M_{l-1}(x,y) = 1) and -infinity otherwise. After the softmax operation, locations masked with -infinity receive zero attention weight, effectively restricting the query to only attend to its predicted foreground region. The predicted masks are binarized with a threshold of 0.5 and used without gradient to define the attention region. For the first decoder layer where no prior prediction exists, the attention mask is initialized from learnable query features that serve as mask proposals. This mechanism creates a feedback loop: better masks lead to better attention regions, which in turn produce better masks.

Compared with alternative approaches to restricting attention, masked attention offers distinct advantages. Deformable attention in Deformable DETR learns to attend to a sparse set of sampling points, but these points may not align well with object boundaries. Mask pooling used in K-Net applies predicted masks to pool features into a single vector per query, losing spatial structure. In contrast, masked attention preserves the spatial resolution of attention while constraining it to relevant regions. Empirically, masked attention achieves 43.7 AP on COCO instance segmentation, compared to 37.8 AP for standard cross-attention, 37.9 AP for deformable attention (SMCA), and 43.1 AP for mask pooling, validating our design choice.

Utilizing high-resolution features is crucial for accurately segmenting small objects and fine boundaries. The pixel decoder produces a feature pyramid at three resolutions: 1/32, 1/16, and 1/8 of the original image. A naive approach would concatenate all resolutions and feed them to every Transformer decoder layer, but this is prohibitively expensive in computation and memory. Instead, we propose an efficient round-robin strategy: each Transformer decoder layer receives features from only one resolution level, cycling through the three scales in succession. We repeat this 3-layer pattern L times, yielding a total of 3L decoder layers (9 layers with L=3).

Each resolution level is augmented with sinusoidal positional embeddings to encode spatial information, plus a learnable scale-level embedding shared across all spatial locations of the same resolution. These scale-level embeddings allow the Transformer to distinguish which resolution it is processing in each layer. Compared to the naive multi-scale approach that concatenates all resolutions (achieving 44.0 AP at 247 GFLOPs), our round-robin strategy maintains comparable performance at 43.7 AP with only 226 GFLOPs, a reduction of approximately 10% in computation. The practical benefit is that high-resolution features can be incorporated without the quadratic cost explosion of full-resolution attention, making the approach scalable to high-resolution inputs.

We introduce three optimization improvements to the Transformer decoder. First, we switch the order of self-attention and cross-attention. In the standard Transformer decoder, self-attention is applied first, followed by cross-attention. However, the query features fed to the first self-attention layer are image-independent (either zero-initialized or learnable parameters), making the initial self-attention ineffective. By placing cross-attention before self-attention, queries first gather image-specific information, after which self-attention can meaningfully model inter-query relationships. Second, we make query features X_0 learnable instead of zero-initialized. These learnable queries are directly supervised before entering the Transformer decoder, functioning as a region proposal network that provides initial mask proposals. This gives the decoder a warm start with reasonable initial predictions rather than starting from blank.

Third, we find that dropout is unnecessary in the Transformer decoder and typically decreases performance for dense prediction tasks. We remove dropout entirely from the decoder, observing consistent improvement across all three segmentation tasks. Beyond architecture changes, we address training memory efficiency through a point-based loss calculation. Instead of computing the mask loss over the entire output mask, we randomly sample K = 12,544 points and compute the loss only at these locations. For the bipartite matching, we use uniform random sampling; for the final training loss, we use importance sampling that oversamples uncertain regions. This point-based approach reduces per-image training memory from 18 GB to 6 GB — a 3x reduction — making it possible to train with multiple images per GPU on standard 32 GB hardware, directly addressing the accessibility limitation of MaskFormer.

We evaluate Mask2Former on four major segmentation benchmarks: COCO, ADE20K, Cityscapes, and Mapillary Vistas. On COCO panoptic segmentation, Mask2Former with a Swin-L backbone achieves 57.8 PQ, surpassing MaskFormer (52.7 PQ) by 5.1 points and K-Net (54.6 PQ) by 3.2 points. Notably, Mask2Former converges in only 50 training epochs compared to MaskFormer's 300 epochs, a 6x improvement in training efficiency. Even with a ResNet-50 backbone, Mask2Former reaches 51.9 PQ, already outperforming many methods that use larger backbones. On ADE20K panoptic segmentation, Mask2Former achieves 48.1 PQ, setting a new state-of-the-art on this challenging dataset.

On COCO instance segmentation, Mask2Former achieves 50.1 AP with a Swin-L backbone, which is the first time a universal architecture outperforms the best specialized instance segmentation model. Specifically, Mask2Former surpasses Swin-HTC++ (49.5 AP), which uses a cascade structure specifically designed for instance segmentation. The improvement is particularly notable on boundary-aware metrics: Mask2Former achieves 2.1 AP higher on AP_boundary, suggesting that masked attention helps produce sharper mask boundaries. On Cityscapes instance segmentation, Mask2Former achieves 43.6 AP, also setting a new state-of-the-art. This result is significant because instance segmentation has been the weakest point of universal architectures, and closing this gap removes the last major argument for task-specific designs.

On ADE20K semantic segmentation, Mask2Former achieves 57.7 mIoU with Swin-L backbone and FaPN pixel decoder, surpassing the previous best result of BEiT-UperNet at 57.0 mIoU. On Cityscapes semantic segmentation, Mask2Former reaches 83.3 mIoU, competitive with specialized methods. On the larger Mapillary Vistas dataset, Mask2Former demonstrates consistent improvements across all three tasks. These results confirm that Mask2Former is not merely competitive but sets new state-of-the-art benchmarks as a single unified architecture across all tasks and datasets. The pixel decoder choice matters for semantic segmentation: the FaPN decoder outperforms MSDeformAttn on semantic tasks (57.7 vs. 56.4 mIoU), while MSDeformAttn is better for instance tasks (43.7 vs. 42.7 AP), suggesting that some task-specific tuning of the pixel decoder may still be beneficial.

We conduct comprehensive ablation studies using a ResNet-50 backbone to analyze each component's contribution. Removing masked attention causes the largest degradation: -5.9 AP on instance segmentation, -4.8 PQ on panoptic segmentation, and -1.7 mIoU on semantic segmentation. Removing high-resolution multi-scale features leads to -2.2 AP, -1.7 PQ, and -1.1 mIoU. Removing learnable queries reduces performance by -0.8 AP, -0.7 PQ, and -1.8 mIoU. Switching back to cross-attention first (original order) costs -0.5 AP, -0.3 PQ, and -0.9 mIoU. These results clearly establish a hierarchy of importance: masked attention contributes the most, followed by multi-scale features, learnable queries, and attention ordering.

A deeper analysis of masked attention reveals its mechanism of improvement. Visualizing the attention maps across decoder layers shows that masked attention quickly focuses on object regions from early layers, while standard cross-attention exhibits diffuse attention patterns even in later layers. The learnable queries as mask proposals generate reasonable initial segmentations: the average recall at 100 proposals (AR@100) improves progressively from layer 0 through layer 9, confirming the iterative refinement behavior. We also analyze the pixel decoder architecture, finding that MSDeformAttn consistently achieves the best overall performance (43.7 AP) while BiFPN is better for instance tasks (43.5 AP) and FaPN is better for semantic tasks (46.8 mIoU). This modularity — where the pixel decoder can be swapped without changing the Transformer decoder — is a practical advantage for adapting Mask2Former to different deployment scenarios.

We have presented Mask2Former, a universal image segmentation architecture built upon masked attention in the Transformer decoder. By constraining cross-attention to predicted mask regions, incorporating multi-scale features efficiently, and applying targeted optimization improvements, a single architecture now performs on par with or better than specialized architectures across panoptic, instance, and semantic segmentation. Mask2Former achieves state-of-the-art results on COCO, ADE20K, Cityscapes, and Mapillary Vistas for every task, while being easy to train — reducing research effort by at least three times. Our work demonstrates that the pursuit of universal architectures need not come at the cost of task-specific performance. We acknowledge that models still require task-specific training; unifying training across multiple segmentation tasks remains an important direction for future work. We hope Mask2Former inspires the community to embrace universal model design, reducing fragmentation across image segmentation research.