Modern approaches to semantic segmentation typically formulate the task as per-pixel classification. The authors argue that mask classification — predicting a set of binary masks each associated with a single class label — is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner. Following this insight, the authors propose MaskFormer, which adopts a Transformer decoder to compute per-segment embeddings that generate class predictions and mask embeddings simultaneously. MaskFormer achieves state-of-the-art results on ADE20K semantic segmentation (55.6 mIoU) and competitive performance on COCO panoptic segmentation (52.7 PQ), demonstrating that a single mask classification model can simplify the landscape of segmentation approaches.

Since the introduction of Fully Convolutional Networks (FCN), per-pixel classification has been the de facto standard for semantic segmentation. This approach assigns a class probability distribution to each pixel independently, which is natural for semantic segmentation where each pixel belongs to exactly one category. However, for instance-level tasks (instance segmentation and panoptic segmentation), per-pixel classification is fundamentally insufficient — these tasks require distinguishing different instances of the same class, which cannot be achieved by independent pixel-level classification. This has led to divergent methodologies: per-pixel classification for semantic segmentation, and detection-based approaches (like Mask R-CNN) for instance segmentation. The authors propose that mask classification can serve as a universal paradigm for all segmentation tasks.

Mask classification formulates segmentation differently: instead of classifying each pixel independently, the method splits the task into (1) partitioning the image into N regions (binary masks) and (2) associating each region with a class distribution over K categories. This formulation naturally handles variable numbers of segments — a requirement for instance segmentation where the number of objects is unknown — while being equally applicable to semantic segmentation where masks can overlap and are resolved via argmax aggregation. The key advantage is that a single model architecture can address semantic, instance, and panoptic segmentation without task-specific modifications.

Per-pixel classification has dominated semantic segmentation since FCN, with subsequent works (DeepLab, PSPNet, SegFormer) all maintaining this paradigm while improving feature extraction. For instance segmentation, Mask R-CNN popularized a detect-then-segment approach, and DETR later introduced a set-prediction framework using Transformer decoder with learnable queries. Panoptic segmentation further demonstrated the limitations of per-pixel classification, requiring complex multi-branch architectures to merge semantic and instance predictions. The mask classification idea draws inspiration from DETR's set prediction approach but applies it to all segmentation tasks, not just instance-level ones.

Rather than assigning class probabilities to each pixel independently, mask classification splits the segmentation task into two sub-problems: (1) partitioning the image into N regions represented as binary masks {m_i}, and (2) associating each region with a probability distribution p_i over K+1 categories (including a "no object" class). The training objective uses Hungarian matching to find the optimal assignment between predicted and ground truth segments, then computes a combination of cross-entropy loss for class predictions and binary cross-entropy plus dice loss for mask predictions. This formulation allows the number of predicted segments N to be independent of the number of categories K, enabling a single model to handle tasks with vastly different numbers of output segments.

MaskFormer consists of three modules. The pixel-level module extracts per-pixel embeddings using a backbone (e.g., ResNet or Swin Transformer) followed by a pixel decoder that gradually upsamples features to generate high-resolution embeddings. The Transformer decoder takes N learnable query embeddings and, through cross-attention with the image features, produces N per-segment embeddings. The segmentation module then generates: (1) class predictions via a linear classifier on segment embeddings, and (2) binary masks via dot product between segment embeddings and per-pixel embeddings followed by sigmoid activation. This design is backbone-agnostic and can leverage any existing feature extractor.

The inference strategy differs by task. For semantic segmentation, MaskFormer uses marginalization: at each pixel, the probability of category c is computed as the sum over all N masks of the product of mask probability and class probability for c, then argmax over categories gives the final label. This allows multiple queries to contribute to the same semantic class, effectively handling large or disjoint regions of the same category. For panoptic and instance segmentation, a simpler argmax over masks assigns each pixel to the most confident segment, with a confidence threshold to filter low-quality predictions. This demonstrates that the same trained model handles different tasks solely by changing the inference procedure.

A key finding is that MaskFormer's advantage over per-pixel baselines increases with the number of categories. On Cityscapes (19 classes), the improvement is minimal. On ADE20K (150 classes), MaskFormer with Swin-Large backbone achieves 55.6 mIoU, establishing state-of-the-art. On ADE20K-Full (847 classes), the gain reaches +3.5 mIoU over the per-pixel baseline, demonstrating that mask classification becomes increasingly advantageous as the number of categories grows. For COCO panoptic segmentation, MaskFormer achieves 52.7 PQ, competitive with specialized panoptic methods. The model handles both things and stuff categories without architectural changes, confirming the unification claim.

MaskFormer demonstrates that mask classification is a viable and often superior alternative to per-pixel classification for semantic segmentation, while naturally extending to instance and panoptic segmentation. The approach achieves state-of-the-art results on ADE20K and competitive performance across multiple benchmarks. The findings suggest that the segmentation community should reconsider the dominance of per-pixel classification and explore mask-based formulations that offer greater flexibility and scalability. The authors envision this as a step toward truly unified segmentation models that can handle any segmentation task with a single architecture.