Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks, trained end-to-end, pixels-to-pixels, on semantic segmentation exceed the state-of-the-art without further machinery. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, adapt contemporary classification networks (AlexNet, VGGNet, GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a skip architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations.

Convolutional networks are driving advances in recognition. Convnets are not only improving for whole-image classification, but also making progress on local tasks with structured output. This includes advances in bounding box object detection, part and keypoint prediction, and local correspondence. The natural next step in the progression from coarse to fine inference is to make a prediction at every pixel. Prior approaches have used convnets for semantic segmentation, but have done so in ways that involve patchwise training, post-processing with superpixels or CRFs, or both.

We show that a fully convolutional network (FCN), trained end-to-end, pixels-to-pixels on semantic segmentation, surpasses the state-of-the-art without further machinery. To our knowledge, this is the first work to train FCNs end-to-end for pixelwise prediction and from supervised pre-training. Fully convolutional versions of existing classification networks predict dense outputs from inputs of any size. Both learning and inference are performed whole-image-at-a-time by dense feedforward computation and backpropagation. In-network upsampling layers enable pixelwise prediction in nets with subsampled pooling.

Semantic segmentation has a long history in computer vision. Current approaches often combine bottom-up recognition with top-down refinement. Deep classification networks have been repurposed for segmentation by applying them in a sliding window fashion, which is computationally expensive and limits the receptive field. Others have used recurrent neural networks or conditional random fields (CRFs) as post-processing to refine coarse predictions. In contrast, our approach is entirely feedforward and does not require any post-processing. The idea of deconvolution for upsampling has appeared before, but not in the context of end-to-end training for dense prediction from full images.

Each layer of data in a convnet is a three-dimensional array of size h x w x d, where h and w are spatial dimensions, and d is the feature or channel dimension. The first layer is the image, with pixel size h x w, and d color channels. Locations in higher layers correspond to the locations in the image they are path-connected to, which are called their receptive fields. A network with only convolutional layers (and pooling) computes a nonlinear filter, referred to as a fully convolutional network. An FCN naturally operates on an input of any size, and produces an output of corresponding spatial dimensions. The classification nets (AlexNet, VGGNet, GoogLeNet) can be cast into fully convolutional form by replacing their fully connected layers with convolutional layers.

The output of the fully convolutional classification network is a coarse map at 1/32 of the input resolution. While this captures high-level semantic information, it lacks fine spatial detail. To address this, we use backwards convolution (deconvolution) with learned filters for upsampling. More importantly, we introduce skip connections that combine predictions from the final layer with those from earlier, finer layers. Our FCN-32s uses only the final prediction layer; FCN-16s fuses predictions from the final and pool4 layer; FCN-8s further adds the pool3 layer. This skip architecture yields a nonlinear feature hierarchy that combines deep, coarse, semantic information with shallow, fine, appearance information for progressively finer predictions.

We test FCN on PASCAL VOC 2011 and 2012 segmentation challenges. FCN-8s achieves 62.2% mean IU on the PASCAL VOC 2012 test set, a 20% relative improvement over the previous state-of-the-art. We also report results on NYUDv2 and SIFT Flow datasets, achieving state-of-the-art on both. Inference takes less than one-fifth of a second per image for a typical 500x500 input, compared to minutes for patch-based approaches. Different base networks yield different quality: VGG-16 as the base network provides the best performance, while GoogLeNet yields slightly lower accuracy. Fine-tuning the entire network end-to-end is critical: freezing layers significantly degrades results.

Fully convolutional networks are a rich class of models that can be trained end-to-end for pixelwise prediction. We have shown that adapting classification networks and applying them fully convolutionally for dense prediction is a simple, effective, and scalable approach. The key insight is the combination of deep, coarse layers with shallow, fine layers through skip connections, enabling predictions that respect both global structure and local detail. We believe that fully convolutional training and prediction will become standard for dense prediction tasks.