Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed PSPNet. Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction. The approach achieves state-of-the-art performance on various datasets. It came first in ImageNet Scene Parsing Challenge 2016, and obtained mIoU accuracy of 85.4% on PASCAL VOC 2012 and 80.2% on Cityscapes.

Scene parsing, based on semantic segmentation, is a fundamental topic in computer vision. The goal is to assign each pixel in an image a category label. Scene parsing provides complete understanding of the scene, which needs to recognize not only individual objects but also the contextual relationships among them. For applications such as autonomous driving, robot navigation, and image editing, scene parsing is extremely important.

Most recent approaches are based on fully convolutional networks (FCN). Despite their effectiveness, FCN-based methods lack the ability to leverage global context information. Pixel-level features from deep CNNs have limited receptive fields, which may result in misclassification of categories that depend on context. For example, a boat is more likely to appear near water rather than on a road. The authors observe three common failure cases: mismatched relationships (ignoring co-occurrence context), confusion categories (visually similar classes like building vs. skyscraper), and inconspicuous classes (very small or very large objects).

Recent advances in pixel-level prediction follow two primary directions: multi-scale feature ensembling and structure prediction approaches. Multi-scale features extracted from different layers carry both local and global information. Methods like DeepLab use atrous spatial pyramid pooling (ASPP) to capture multi-scale context. However, these global descriptors remain insufficient for complex datasets like ADE20K with 150 categories. The authors differentiate PSPNet by performing "different-region-based context aggregation via our pyramid scene parsing network" rather than simple global pooling.

The authors identify that while theoretical receptive fields in deep networks like ResNet exceed input dimensions, empirical receptive fields are actually much smaller, limiting the network's ability to incorporate global context. This gap between theoretical and effective receptive fields means that even very deep networks may fail to capture scene-level information. The three failure modes identified — mismatched relationships, confusion categories, and inconspicuous classes — all point to the same root cause: insufficient exploitation of global context prior.

The pyramid pooling module fuses features under four different pyramid scales. The module uses bin sizes of 1x1, 2x2, 3x3, and 6x6 to divide the feature map into sub-regions. At each pyramid level: (1) average pooling is applied to generate sub-region representations; (2) a 1x1 convolution reduces the channel dimension to 1/N of the original (where N is the number of pyramid levels); (3) bilinear interpolation upsamples the output back to the original feature map size. The outputs from all levels are concatenated with the original feature map to form the final pyramid pooling global prior representation. This hierarchical pooling captures context at multiple granularities — from global scene statistics (1x1) to local region patterns (6x6).

The network architecture uses a pretrained ResNet with the dilated convolution strategy to produce feature maps at 1/8 of the input resolution. The pyramid pooling module is applied on top of these feature maps to aggregate context information. The concatenated features are then fed into a final convolution layer to generate pixel-wise predictions. Compared to approaches using global average pooling alone, this design provides a richer, hierarchical representation of global context. The authors show that average pooling outperforms max pooling across all experiments.

Deep networks face optimization difficulties despite skip connections. The authors propose an auxiliary loss applied after the res4b22 residual block in ResNet-101, generating an initial supervised result. Both the auxiliary branch loss and the master branch loss propagate through all preceding layers. During training, the auxiliary loss is weighted by 0.4, with the master branch assuming primary responsibility. During testing, only the master branch performs prediction, avoiding additional computational overhead. This deep supervision strategy improves the ResNet-50 baseline by 1.41 points in Mean IoU.

Extensive experiments are conducted on ImageNet Scene Parsing (ADE20K), PASCAL VOC 2012, and Cityscapes. On ADE20K, the best single model achieves 44.94% Mean IoU with ResNet-269 and multi-scale testing, while the ensemble submission reaches 57.21% (first place). Ablation studies demonstrate that: (1) multi-level pyramids outperform single global pooling by 1.61 Mean IoU points; (2) auxiliary loss optimization improves the baseline by 1.41 points; (3) deeper ResNet models consistently yield improvements. On PASCAL VOC 2012, PSPNet achieves 85.4% mIoU with MS-COCO pretraining, outperforming competing approaches without CRF post-processing. On Cityscapes, it reaches 80.2% IoU using both fine and coarse annotations.

PSPNet provides an effective framework for complex scene understanding by combining global pyramid pooling features with local FCN representations. The pyramid pooling module captures hierarchical global context at multiple scales, directly addressing the limitations of fixed receptive fields in standard FCN architectures. Together with the deep supervision optimization strategy, PSPNet achieves state-of-the-art performance across multiple challenging benchmarks. The work includes practical implementation details to enable community adoption for semantic segmentation and related pixel-level prediction tasks.