The authors propose a novel semantic segmentation approach using a learned deconvolution network built on top of VGG 16-layer convolutional features. Unlike FCN-based methods that produce coarse segmentation maps due to fixed receptive fields, this approach applies the deconvolution network to individual object proposals and then combines the results to construct the final pixel-wise segmentation. The method handles objects at multiple scales naturally and captures fine-grained structural details, achieving 72.5% mean IoU on PASCAL VOC 2012 when ensembled with FCN — the best result among methods not using external training data.

Semantic segmentation — assigning a class label to every pixel — is fundamental to scene understanding. Fully Convolutional Networks (FCN) have become the dominant approach, yet they suffer from two critical limitations. First, the fixed-size receptive field makes it difficult to handle objects at varying scales: large objects are only partially activated, while small objects are often misclassified as background. Second, the coarse label maps produced by FCN (typically 16x16) lose fine structural details, and the simple bilinear interpolation used for upsampling cannot recover them.

To address both limitations, the authors propose a deconvolution network that mirrors the convolutional architecture with symmetric deconvolution and unpooling layers. Rather than processing the entire image at once, the network is applied to individual object proposals, which naturally provides scale invariance since each proposal is resized to a canonical input size. The deconvolution layers progressively reconstruct the shape of objects from coarse to fine, with lower layers capturing overall shapes and higher layers encoding class-specific fine details.

Fully Convolutional Networks (FCN) by Long et al. established the paradigm of converting classification networks into dense prediction models by replacing fully connected layers with convolutional layers. DeepLab combined FCN features with dense CRF post-processing for boundary refinement. Encoder-decoder architectures have emerged as a general framework, where the encoder extracts features through progressive downsampling and the decoder reconstructs spatial resolution. The proposed deconvolution network contributes to this line of work by introducing learned unpooling and deconvolution operations rather than relying on fixed bilinear interpolation or simple skip connections.

The deconvolution network is composed of two parts: a convolution network (encoder) identical to VGG-16 that progressively reduces spatial resolution, and a mirrored deconvolution network (decoder) that progressively restores it. The decoder consists of unpooling, deconvolution, and ReLU layers arranged symmetrically with the encoder. The key innovation is that unpooling layers use the pooling indices recorded during the encoding phase to place activations at the exact original locations, thereby preserving spatial information that is lost in standard upsampling.

Unpooling performs the reverse of max pooling: it records the locations of maximum activations during pooling (switch variables) and uses these locations to place activations in the upsampled feature map. All non-maximum positions are filled with zeros, creating a sparse activation map that preserves the spatial structure of the input. Deconvolution layers then densify these sparse maps through learned filters: unlike standard convolution where many inputs contribute to one output, in deconvolution a single input activation is associated with multiple outputs, effectively enlarging the feature map. The hierarchical structure means that lower deconvolution layers capture overall object shape while higher layers reconstruct class-specific fine details.

Training proceeds in two stages. In Stage 1, the network is trained on ground-truth centered object crops, which reduces the search space and requires approximately 0.2M training examples. In Stage 2, training shifts to object proposals with variable positioning, creating robustness to localization noise, using approximately 2.7M examples. Batch normalization is applied throughout the network to address the internal covariate shift problem and stabilize training of this deep architecture. During inference, approximately 2,000 object proposals are generated per image using edge-box methods, the top 50 are selected by objectness score, and outputs are aggregated via pixel-wise maximum operations.

Evaluation on the PASCAL VOC 2012 test set demonstrates the effectiveness of the approach. The DeconvNet alone achieves 69.6% mean IoU, improving to 70.5% with CRF post-processing. The ensemble of DeconvNet and FCN (EDeconvNet+CRF) achieves 72.5% mean IoU, which is the best result among methods not using external training data. This outperforms FCN-8s (62.2%) by over 10 points and DeepLab-CRF (71.6%) by nearly 1 point. The ensemble works because the two methods are complementary: DeconvNet excels at fine object details while FCN captures overall shapes more effectively. Simple averaging of probability maps from both approaches yields the best results.

This paper presents a deconvolution network that effectively addresses the limitations of FCN-based semantic segmentation. The architecture reconstructs object structures through progressive coarse-to-fine deconvolution operations, using learned unpooling with recorded switch variables to preserve spatial precision. The instance-wise prediction via object proposals provides natural scale handling. Combined with FCN through simple ensemble averaging, the method achieves state-of-the-art performance on PASCAL VOC 2012. The complementary nature of the two approaches suggests that both holistic scene understanding and fine-grained instance analysis are important for accurate semantic segmentation.