Deep residual networks have emerged as a family of extremely deep architectures showing compelling accuracy and nice convergence behaviors. In this paper, we analyze the propagation formulations behind the residual building blocks, which suggest that the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation. A series of ablation experiments support the importance of these identity mappings. This motivates us to propose a new residual unit, which makes training easier and improves generalization. We report improved results using a 1001-layer ResNet on CIFAR-10 (4.62% error) and CIFAR-100, and a 200-layer ResNet on ImageNet. Code is available at: https://github.com/KaimingHe/resnet-1k-layers.

Deep residual networks (ResNets) consist of many stacked "Residual Units". Each unit can be expressed in a general form: y_l = h(x_l) + F(x_l, W_l), where x_l and x_{l+1} are input and output of the l-th unit, and F is a residual function. In the original work, h(x_l) = x_l is an identity mapping and f is a ReLU function. ResNets that are over 100-layer deep have shown state-of-the-art accuracy for several challenging recognition tasks on ImageNet and MS COCO competitions. The central idea of ResNets is to learn the additive residual function F with respect to h(x_l), with a key choice of using an identity mapping h(x_l) = x_l. This is realized by attaching an identity skip connection.

In this paper, we analyze deep residual networks by focusing on creating a "direct" path for propagating information — not only within a residual unit, but through the entire network. Our derivations reveal that if both h(x_l) and f(y_l) are identity mappings, the signal could be directly propagated from one unit to any other units, in both forward and backward passes. Our experiments empirically show that training in general becomes easier when the architecture is closer to the above two conditions. To understand the role of skip connections, we analyze and compare various types of h(x_l). We find that the identity mapping h(x_l) = x_l chosen in the original ResNet achieves the fastest error reduction and lowest training loss among all variants we investigated.

The ResNets developed in the original work are modularized architectures that stack building blocks of the same connecting shape. In this paper we call these blocks "Residual Units". The original Residual Unit performs the following computation: y_l = h(x_l) + F(x_l, W_l), x_{l+1} = f(y_l). With the recursive computation of x_L = x_l + sum of F, for any deeper unit L and any shallower unit l. This equation exhibits some nice properties. (i) The feature x_L of any deeper unit L can be represented as the feature x_l of any shallower unit l plus a residual function, indicating that the model is in a residual fashion between any units L and l. (ii) The feature x_L = x_0 + sum of all residual functions, which is in contrast to a "plain network" where a feature x_L is a series of matrix-vector products.

Let us consider a simple modification, h(x_l) = lambda_l * x_l, to break the identity shortcut. We investigate various combinations and find that all shortcut-only gating and 1x1 convolutional shortcuts lead to higher training error and higher testing error. These experiments suggest that keeping a "clean" information path is helpful for easing optimization. We also study the impact of the activation functions. The original design uses post-addition activation: x_{l+1} = ReLU(y_l). We propose using pre-activation: y_l = x_l + F(BN(ReLU(x_l)), W_l). In the new design, the signals among the building blocks are clean identity mappings.

We conduct experiments on CIFAR-10, CIFAR-100, and ImageNet. Our baseline ResNets are the 110-layer and 164-layer networks on CIFAR, and the 101-layer network on ImageNet. On CIFAR-10, our 1001-layer pre-activation ResNet achieves 4.62% test error, which is significantly better than the original ResNet's 7.61%. The improvement becomes more pronounced as the network goes deeper. On CIFAR-100, the trend is similar — our 1001-layer network achieves 22.71% error. On ImageNet, our 200-layer pre-activation ResNet achieves a top-1 error of 21.66%, which is lower than the baseline 34-layer plain net's 28.54%. The pre-activation design also demonstrates improved generalization: the gap between training and testing error is smaller.

We also investigate the activation function placement in detail. We compare: (a) original post-activation (BN after addition), (b) BN before addition, (c) ReLU before addition, (d) ReLU-only pre-activation, and (e) full pre-activation (BN + ReLU before weight layers). The full pre-activation variant achieves the best results. The key insight is that the BN layer as pre-activation improves the regularization of the model, since it normalizes the signal flowing through the shortcut. This is evidenced by the reduced overfitting observed in our experiments.

This paper investigates the propagation formulations behind the connection mechanisms of deep residual networks. Our derivations imply that identity shortcut connections and identity after-addition activation are essential for making information propagation smooth. Our proposed architecture with pre-activation residual units facilitates training of very deep networks (over 1000 layers) and improves generalization. We believe that exploring the propagation properties will give further insight for the design of deep networks.