We address the problem of semantic segmentation with a simple yet effective approach called Object-Contextual Representations (OCR). The key idea is to explicitly represent the object region context and augment the pixel representation with object-contextual information. First, we learn soft object regions from ground-truth segmentation. Then we compute object region representations by aggregating pixel representations within each region. Finally, we augment pixel representations with object-contextual representations computed as a weighted aggregation of all object region representations. OCR achieves state-of-the-art performance on Cityscapes (84.5% mIoU), ADE20K (45.28% mIoU), and LIP (55.60% mIoU).

The key challenge for semantic segmentation is learning good pixel representations. Current approaches rely on multi-scale context aggregation like ASPP (Atrous Spatial Pyramid Pooling) or PPM (Pyramid Pooling Module). These methods aggregate context from a fixed spatial range regardless of object structure, mixing representations from different categories and leading to suboptimal representations near boundaries. We argue that context should be object-aware: each pixel should be influenced by representations from the same object category.

Our OCR approach characterizes pixels by exploiting corresponding object class representations. We first compute a coarse segmentation to identify object regions, then compute per-category object representations by aggregating features, and finally update each pixel with a weighted combination of object representations. Weights are computed based on pixel-object similarity, ensuring each pixel attends to the most relevant objects.

The OCR module consists of three stages. The Soft Object Region stage computes a coarse segmentation map using an auxiliary head, providing soft pixel-to-class assignments. The Object Region Representation stage aggregates pixel representations weighted by soft assignments to obtain K object region representations. The Object Contextual Representation stage computes pixel-to-region similarities and augments each pixel with a weighted sum of object representations.

Formally, let f_k denote the object representation for the k-th class, computed as a weighted average of pixel features. The object-contextual representation for pixel i is a weighted sum of all K representations, where weights are determined by normalized dot-product similarity. This mechanism allows each pixel to selectively attend to the most relevant object context, naturally handling multi-scale objects and complex boundaries. Compared to standard self-attention (O(N^2)), OCR reduces complexity to O(NK) where K is the number of categories, making it much more efficient.

On Cityscapes test, OCR achieves 84.5% mIoU with HRNet-W48, setting a new state of the art. On ADE20K validation, OCR reaches 45.28% mIoU. On LIP validation, 55.60% mIoU. Compared to baselines with the same backbone, OCR consistently provides +1 to +2% mIoU improvement with less than 5% additional FLOPs.

Ablation studies confirm each component's contribution. Replacing OCR with standard self-attention reduces mIoU by 0.8%. Removing auxiliary segmentation loss reduces mIoU by 0.5%. The soft region assignment outperforms hard assignment by 0.3%, validating the design choice of using probabilistic rather than deterministic object regions.

We have presented OCR for semantic segmentation, a simple approach that explicitly leverages object-level context to improve pixel representations. OCR achieves state-of-the-art results with minimal overhead. We believe object-aware context aggregation is a principled direction for semantic segmentation, and even simple implementations yield significant improvements.