Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs). We demonstrate that a 16-layer-deep wide residual network has comparable or better accuracy than very deep thin networks (over 1000 layers), while being several times faster to train. We achieve state-of-the-art results on CIFAR-10 (3.89%), CIFAR-100 (18.85%), and SVHN (1.54%).

Since the introduction of ResNets, the dominant trend in neural network design has been to make networks deeper. ResNets with 100, 200, and even 1000 layers have been explored. However, we observe that the benefits of going deeper diminish rapidly, and much of the representational capacity in very deep networks may be wasted. In contrast, increasing the width (number of channels) of residual blocks is a more computationally efficient way to improve performance. Wider networks also have the advantage of being more amenable to parallelization on modern GPUs, since matrix multiplications with wider layers can better utilize GPU cores.

We parameterize the width of residual blocks by a widening factor k. A standard ResNet with n layers has channel widths of [16, 32, 64]; with k=10, these become [160, 320, 640]. We systematically vary both depth (d) and width (k) to study their interaction. Our key finding: WRN-28-10 (28 layers, width factor 10) achieves 3.89% error on CIFAR-10, outperforming a 1001-layer pre-activation ResNet (4.62%) while being 8x faster to train. Further, WRN-16-8 achieves similar accuracy to the 1001-layer network but trains 10x faster. These results strongly suggest that width is a more effective dimension to increase than depth beyond a certain point.

Wider networks have more parameters and may be prone to overfitting. We study the use of dropout between convolutional layers within residual blocks as a regularization technique. We insert dropout between the two 3x3 convolutions in each residual block. Our experiments show that dropout with rate 0.3-0.4 consistently improves performance for wide networks. Interestingly, dropout does not help thin (standard-width) networks, suggesting that regularization is particularly important when increasing width. The combination of widening with dropout achieves the best results across all datasets.

We evaluate WRN on CIFAR-10, CIFAR-100, SVHN, and ImageNet. On CIFAR-10, WRN-28-10 achieves 3.89% error, the best result at the time. On CIFAR-100, WRN-28-10 achieves 18.85% error. On SVHN, WRN-16-8 achieves 1.54%. Training time comparisons show WRNs are significantly faster to train than very deep networks with similar accuracy. On ImageNet, WRNs also show competitive performance. We provide comprehensive ablation studies covering depth, width, dropout rate, and their interactions.

We have conducted a thorough experimental study of residual network architectures and demonstrated that wide residual networks are a simple, efficient, and effective way to improve the performance of residual networks. Our results challenge the prevailing wisdom that deeper is always better, showing that increasing width with appropriate regularization can achieve superior results to increasing depth, while being much faster to train.