Convolutional networks are at the core of most state-of-the-art computer vision solutions for a wide variety of tasks. Since 2014 very deep convolutional networks started to become mainstream, yielding substantial gains in various benchmarks. Although increased model size and computational cost tend to translate to immediate quality gains, computational efficiency and low parameter count are still enabling factors for various use cases. Here we explore ways to scale up networks in ways that aim at utilizing the added computation as efficiently as possible by suitably factorized convolutions and aggressive regularization. We benchmark our methods on the ILSVRC 2012 classification challenge and achieve 21.2% top-1 and 5.6% top-5 error for single frame evaluation using a network with a computational cost of 5 billion multiply-adds per inference and with under 25 million parameters.

Since the 2012 ImageNet competition, improvements in convolutional neural network quality have been leveraged across a wide variety of applications, from object detection to image segmentation to action recognition. While VGGNet offered the appealing feature of architectural simplicity, this comes at a high cost: evaluating the network requires a lot of computation. The Inception architecture of GoogLeNet was designed to perform well even under strict constraints on memory and computational budget, achieving efficiency with "only 5 million parameters, which represented a 12x reduction" compared to AlexNet. However, the complexity of the Inception architecture makes it more difficult to adapt and modify the network. In this paper, we provide several design principles and optimization techniques to scale Inception-style networks efficiently.

We propose four general design principles for convolutional network architectures. Principle 1: Avoid representational bottlenecks, especially early in the network. The "representation size should gently decrease from the inputs to the outputs". Principle 2: Higher dimensional representations are easier to process locally. Increasing the activations per tile allows for "more disentangled features" and faster training. Principle 3: Spatial aggregation can be done over lower dimensional embeddings without much loss in representational power, since adjacent units have strong correlations. Principle 4: Balance the width and depth of the network. The optimal improvement is achieved by increasing both width and depth in parallel, distributing the computational budget proportionally.

A key technique for improving efficiency is factorizing larger convolutions into smaller ones. A 5x5 convolution can be replaced by two sequential 3x3 convolutions, achieving 28% computational savings while maintaining the same receptive field. Furthermore, spatial convolutions can be factorized into asymmetric pairs: a 3x3 convolution decomposed into a 3x1 followed by 1x3 provides 33% savings versus standard 3x3 operations, which is superior to the 11% savings from 2x2 factorization. However, asymmetric factorization does not work well on early layers and is most effective at medium grid sizes (feature maps of 12-20). On such grids, very good results can be achieved using 1xn followed by nx1 filters.

The original GoogLeNet used auxiliary classifiers attached to intermediate layers, hypothesized to combat vanishing gradients and provide additional regularization. Our experiments reveal a surprising finding: auxiliary classifiers do not improve convergence early in training but act as regularizers. The effect "emerges near the end of training" when the network with auxiliary branches starts to overtake the one without. Furthermore, adding batch normalization to the auxiliary classifier branch yields an additional 0.4% absolute gain in top-1 accuracy, suggesting that batch normalization acts as a regularizer in this context as well.

A common issue in network design is how to reduce the spatial resolution without creating a representational bottleneck. Applying pooling before expanding channels causes a bottleneck, while expanding before pooling is computationally expensive. The solution is to use parallel stride-2 blocks: a pooling path and a convolution path operate simultaneously, and their results are filter-concatenated. This allows efficient reduction of the grid size while preserving representational richness, following the principle of avoiding bottlenecks.

We propose label smoothing, a mechanism to regularize the classifier layer by replacing the hard target distribution with a mixture of the original ground-truth distribution and a uniform distribution. The smoothed target becomes q'(k) = (1 - epsilon) * delta(k, y) + epsilon / K, where epsilon = 0.1 for K = 1000 classes. This prevents the network from becoming too confident in its predictions, which can lead to overfitting. Label smoothing provides a consistent 0.2% absolute improvement for both top-1 and top-5 metrics.

The proposed Inception-v3 architecture is evaluated on ILSVRC 2012. Models are trained using RMSProp optimizer with 0.9 decay, learning rate 0.045 exponentially decayed every two epochs at rate 0.94, gradient clipping at 2.0, across 50 GPU replicas for 100 epochs. Single-crop evaluation achieves 21.2% top-1 error and 5.6% top-5 error with 4.8 billion multiply-add operations. With multi-crop evaluation and an ensemble of 4 models, the error drops to 17.3% top-1 and 3.5% top-5. The architecture uses 42 layers with only 2.5x computational increase versus prior BN-Inception while significantly outperforming competing approaches. Networks with lower resolution inputs (79x79, 151x151, 299x299) achieve comparable results (75.2%, 76.4%, 76.6% top-1) at equivalent computational costs, demonstrating the principles' robustness.

We have provided several design principles to scale convolutional networks efficiently, grounded in factorized convolutions, balanced network dimensions, and aggressive regularization. The resulting Inception-v3 architecture achieves state-of-the-art results on ILSVRC 2012 with relatively modest computational cost. Our design principles are not specific to the Inception architecture but can inform the design of other network families. The label smoothing regularization technique is broadly applicable and consistently improves performance.