Large Convolutional Network models have recently demonstrated impressive classification performance on the ImageNet benchmark. However there is no clear understanding of why they perform so well, or how they might be improved. In this paper we address both issues. We introduce a novel visualization technique that gives insight into the function of intermediate feature layers and the operation of the classifier. Used in a diagnostic role, these visualizations allow us to find model architectures that outperform Krizhevsky et al. on the ImageNet classification benchmark.

Since their introduction by LeCun et al. in the early 1990s, Convolutional Networks (convnets) have demonstrated excellent performance at tasks such as hand-written digit classification and face detection. In the last year, several papers have shown that they can also deliver outstanding performance on more challenging visual classification tasks. Krizhevsky et al. achieved a convincing win in the ImageNet 2012 classification challenge, with their large convnet achieving significant improvements over previous state-of-the-art.

There are several factors behind this renewed interest in convnet models: (i) the availability of much larger training sets with millions of labeled examples; (ii) powerful GPU implementations that make the training of very large models practical; and (iii) better model regularization strategies, such as Dropout. Despite this progress, there is still little insight into the internal operation and behavior of these complex models, or how they achieve such good performance. Without this understanding, the development of better models is reduced to trial-and-error.

We use a multi-layered Deconvolutional Network (deconvnet) to project the feature activations back to the input pixel space. A deconvnet can be thought of as a convnet model that uses the same components (filtering, pooling) but in reverse, so instead of mapping pixels to features, it maps features to pixels. To examine a given convnet activation, we set all other activations in the layer to zero and pass the feature maps as input to the attached deconvnet layer. Then we successively (i) unpool, (ii) rectify, and (iii) filter to reconstruct the activity in the layer beneath.

Using our visualization technique, we reveal the hierarchical nature of features in a convnet. Layer 1 and 2 respond to low-level features such as edges, corners, and color conjunctions. Layer 3 captures more complex textures and patterns. Layer 4 shows significant variation and begins to capture class-specific features (e.g., dog faces, bird legs). Layer 5 captures entire objects with significant pose variation. This progression from simple to complex features across layers confirms the intuition that deep networks learn increasingly abstract representations.

Guided by the visualization, we modified the architecture of Krizhevsky et al. by (i) reducing the first layer filter size from 11x11 to 7x7 and (ii) reducing the stride from 4 to 2. These changes were motivated by observing that the first layer filters of the original model contained a mixture of high and low frequency information with aliasing artifacts. Our modified architecture achieves a top-5 error rate of 14.8% on ImageNet 2012 validation set, compared to 16.4% for Krizhevsky et al.

We have explored large convnet models using a novel visualization technique that reveals the hierarchical feature representations learned at each layer. The visualizations provide a non-trivial understanding of what a convnet learns and identify potential issues such as aliasing in the first layer. Guided by these visualizations, we constructed an architecture (ZFNet) that achieves state-of-the-art results on several benchmarks. Our work demonstrates that understanding these models is essential for building better ones.