Spatial pyramid pooling module or encode-decoder structure are used in deep neural networks for semantic segmentation task. The former networks are able to encode multi-scale contextual information by probing the incoming features with filters or pooling operations at multiple rates and multiple effective fields-of-view, while the latter networks can capture sharper object boundaries by gradually recovering the spatial information.

In this work, we propose to combine the advantages from both methods. Specifically, our proposed model, DeepLabv3+, extends DeepLabv3 by adding a simple yet effective decoder module to refine the segmentation results especially along object boundaries. We further explore the Xception model and apply depthwise separable convolution to both Atrous Spatial Pyramid Pooling and decoder modules, resulting in a faster and stronger encoder-decoder network.

We demonstrate the effectiveness of the proposed model on the PASCAL VOC 2012 and Cityscapes datasets, achieving the test set performance of 89.0% and 82.1% without any post-processing. Our paper is accompanied with a publicly available reference implementation of the proposed models in TensorFlow.

Semantic segmentation with the goal of assigning a semantic label to every pixel in an image is one of the fundamental topics in computer vision. Deep convolutional neural networks based on the Fully Convolutional Neural Network show striking improvement over systems relying on hand-crafted features on benchmark tasks. In this work, we consider two types of neural networks that use spatial pyramid pooling module or encoder-decoder structure for semantic segmentation, where the former captures rich contextual information by pooling features at different resolution while the latter is able to obtain sharp object boundaries.

In order to combine the advantages of both approaches, we propose to add a decoder module to the existing encoder module, DeepLabv3, to form an encoder-decoder architecture. The rich semantic information is encoded in the output of DeepLabv3, while the detailed object boundaries are recovered by the simple yet effective decoder module. The encoder module allows us to extract features at an arbitrary resolution by applying atrous convolution, and the decoder module refines the segmentation results along object boundaries.

We further explore the Xception model and apply depthwise separable convolution to both ASPP and decoder modules, resulting in a faster and stronger encoder-decoder network. We demonstrate the effectiveness of the proposed model on two competitive benchmark datasets: PASCAL VOC 2012 and Cityscapes. The proposed model, called DeepLabv3+, attains new state-of-the-art performance on both datasets without DenseCRF post-processing.

Models based on Fully Convolutional Networks (FCNs) have demonstrated significant improvements for semantic segmentation. Several model variants have been proposed to exploit the contextual information for segmentation, including those that employ multi-scale inputs or those that adopt probabilistic graphical models such as CRFs. In this work, we mainly discuss four types of FCN-based approaches that are most closely related to our model: atrous convolution, spatial pyramid pooling, encoder-decoder, and depthwise separable convolution.

Atrous convolution, also known as dilated convolution, allows us to repurpose ImageNet pretrained networks to extract denser feature maps by removing the downsampling operations from the last few layers and upsampling the corresponding filter kernels. It is an alternative to using deconvolution layers and offers the advantage of being able to control the resolution of features computed by deep CNNs and adjust the filter's field-of-view in order to capture multi-scale information.

Atrous convolution is a powerful tool that allows us to explicitly control the resolution of features computed by deep neural networks and adjust filter's field-of-view in order to capture multi-scale information, generalizing standard convolution operation. In the case of two-dimensional signals, for each location i on the output feature map y and a convolution filter w, atrous convolution is applied over the input feature map x as: y[i] = sum_k x[i + r * k] * w[k], where the atrous rate r determines the stride with which we sample the input signal.

We employ DeepLabv3 as the encoder module in our proposed encoder-decoder structure. DeepLabv3 employs atrous convolution to extract the features computed by deep convolutional neural networks at an arbitrary resolution. Specifically, we apply Atrous Spatial Pyramid Pooling (ASPP) module which probes convolutional features at multiple scales by applying atrous convolution with different rates. The proposed decoder module consists of 1x1 convolution to reduce the number of channels of the low-level features, followed by concatenation with the upsampled encoder features, and a few 3x3 convolutions to refine the features.

The Xception model has shown promising image classification results on ImageNet with fast computation. More recently, the MSRA team modified the Xception model, called Aligned Xception, and further pushed the performance. Motivated by these findings, we adapt the Xception model for the task of semantic segmentation. In particular, we make several changes: (1) deeper Xception same as in the MSRA version except that we do not modify the entry flow network structure; (2) all max pooling operations are replaced by depthwise separable convolutions with striding; (3) extra batch normalization and ReLU activation are added after each 3x3 depthwise convolution.

We employ ImageNet-1k pretrained ResNet-101 or modified aligned Xception as network backbones for our DeepLabv3+ model. Our implementation is built on TensorFlow. We evaluate the proposed model on the PASCAL VOC 2012 semantic segmentation benchmark which contains 20 foreground object classes and one background class. The original dataset contains 1,464 (train), 1,449 (val), and 1,456 (test) pixel-level annotated images. We augment the dataset by the extra annotations provided by Hariharan et al., resulting in 10,582 (trainaug) training images.

Using the proposed encoder-decoder structure, DeepLabv3+ with the modified Xception as network backbone achieves the performance of 87.8% on the PASCAL VOC 2012 val set and 89.0% on the test set. Compared with other state-of-the-art models, our DeepLabv3+ outperforms PSPNet, Large Kernel Matters, and Multipath-RefineNet. When employing the depthwise separable convolution, the model is significantly faster (multiply-adds are reduced by 33% to 41%) while maintaining similar or better accuracy.

We also evaluate DeepLabv3+ on the Cityscapes dataset, a large-scale dataset that contains 5,000 high quality pixel-level annotated images collected in street scenes from 50 different cities. Our best model achieves 82.1% on the test set, setting a new state-of-the-art. We note that we do not employ DenseCRF post-processing or multi-scale testing for our submissions, which may further improve the performance.

Our proposed model, DeepLabv3+, employs the encoder-decoder structure where DeepLabv3 is used to encode the rich contextual information and a simple yet effective decoder module is adopted to recover the object boundaries. One could also apply the atrous convolution to extract the encoder features at an arbitrary resolution, depending on the available computation resources. We also explore the Xception model and atrous separable convolution to make the proposed model faster and stronger.

Finally, our experimental results show that the proposed model achieves state-of-the-art performance on both PASCAL VOC 2012 and Cityscapes datasets. Our combination of encoder-decoder architecture with atrous separable convolution provides a principled approach that balances accuracy and efficiency for semantic segmentation. We hope our simple yet effective method could serve as a strong baseline for future research in the field.