Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.

On the ImageNet dataset, we evaluate residual nets with a depth of up to 152 layers — 8× deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1,000 layers.

Deep convolutional neural networks have led to a series of breakthroughs for image classification. Deep networks naturally integrate low/mid/high-level features and classifiers in an end-to-end multi-layer fashion, and the "levels" of features can be enriched by the number of stacked layers (depth). Recent evidence reveals that network depth is of crucial importance, and the leading results on the challenging ImageNet dataset all exploit "very deep" models, with a depth of sixteen to thirty layers.

Driven by the significance of depth, a question arises: "Is learning better networks as easy as stacking more layers?" An obstacle to answering this question was the notorious problem of vanishing/exploding gradients, which hamper convergence from the beginning. This problem has been largely addressed by normalized initialization and intermediate normalization layers. When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting, and adding more layers to a suitably deep model leads to higher training error.

In this paper, we address the degradation problem by introducing a deep residual learning framework. Instead of hoping each few stacked layers directly fit a desired underlying mapping H(x), we explicitly let these layers fit a residual mapping F(x) := H(x) − x. The original mapping is recast into F(x) + x. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping. To the extreme, if an identity mapping were optimal, it would be easier to push the residual to zero than to fit an identity mapping by a stack of nonlinear layers.

Residual Representations. In image recognition, VLAD is a representation that encodes by the residual vectors with respect to a dictionary, and Fisher Vector can be formulated as a probabilistic version of VLAD. Both are powerful shallow representations for image retrieval and classification. In low-level vision, solving Partial Differential Equations (PDEs), the Multigrid method reformulates the system as subproblems at multiple scales, where each subproblem is responsible for the residual solution between a coarser and a finer scale. These solvers converge much faster than standard solvers that are unaware of the residual nature of the solutions.

Shortcut Connections. Practices and theories of shortcut connections have been studied for a long time. An early practice of training multi-layer perceptrons is to add a linear layer connected from the network input to the output. Intermediate layers are directly connected to auxiliary classifiers for addressing vanishing/exploding gradients. Concurrent with this work, "highway networks" present shortcut connections with gating functions. These gates are data-dependent and have parameters, in contrast to our identity shortcuts that are parameter-free. When a gated shortcut is "closed," the layers in highway networks represent non-residual functions. On the contrary, our formulation always learns residual functions; our identity shortcuts are never closed, and all information is always passed through.

Let us consider H(x) as an underlying mapping to be fit by a few stacked layers (not necessarily the entire net), with x denoting the inputs to the first of these layers. If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions, then it is equivalent to hypothesize that they can asymptotically approximate the residual functions, i.e., H(x) − x. So rather than expect stacked layers to approximate H(x), we explicitly let these layers approximate a residual function F(x) := H(x) − x. The original function thus becomes F(x) + x. Although both forms should be able to asymptotically approximate the desired functions, the ease of learning might differ.

This reformulation is motivated by the counterintuitive degradation problem. If the added layers can be constructed as identity mappings, a deeper model should have training error no greater than its shallower counterpart. The degradation problem suggests that solvers might have difficulties in approximating identity mappings by multiple nonlinear layers. With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings. In real cases, it is unlikely that identity mappings are optimal, but our reformulation may help to precondition the problem. If the optimal function is closer to an identity mapping than to a zero mapping, it should be easier for the solver to find the perturbations with reference to an identity mapping.

We adopt residual learning to every few stacked layers. A building block is defined as: y = F(x, {W_i}) + x. Here x and y are the input and output vectors of the layers considered. The function F(x, {W_i}) represents the residual mapping to be learned. For the example of two layers, F = W₂σ(W₁x) in which σ denotes ReLU and the biases are omitted for simplifying notations. The operation F + x is performed by a shortcut connection and element-wise addition. We adopt the second nonlinearity after the addition (i.e., σ(y)).

The shortcut connections in the above equation introduce neither extra parameters nor computation complexity. This is not only attractive in practice but also important in our comparisons between plain and residual networks. We can fairly compare plain/residual networks that simultaneously have the same number of parameters, depth, width, and computational cost (except for the negligible element-wise addition). The dimensions of x and F must be equal in the equation. If this is not the case, we can perform a linear projection W_s by the shortcut connections to match the dimensions: y = F(x, {W_i}) + W_sx.

Plain Network. Our plain baselines are mainly inspired by the philosophy of VGG nets. The convolutional layers mostly have 3×3 filters and follow two simple design rules: (i) for the same output feature map size, the layers have the same number of filters; and (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer. We perform downsampling directly by convolutional layers that have a stride of 2. The network ends with a global average pooling layer and a 1000-way fully-connected layer with softmax. The total number of weighted layers is 34. It is worth noting that our model has fewer filters and lower complexity than VGG-19. Our 34-layer baseline has 3.6 billion FLOPs, which is only 18% of VGG-19 (19.6 billion FLOPs).

Residual Network. Based on the above plain network, we insert shortcut connections which turn the network into its residual version. The identity shortcuts can be directly used when the input and output are of the same dimensions. When the dimensions increase, we consider two options: (A) The shortcut still performs identity mapping, with extra zero entries padded for increasing dimensions. This option introduces no extra parameter. (B) The projection shortcut is used to match dimensions (done by 1×1 convolutions). For both options, when the shortcuts go across feature maps of two sizes, they are performed with a stride of 2.

We first evaluate 18-layer and 34-layer plain nets. The 34-layer plain net has higher validation error than the 18-layer plain net. To reveal the reasons, we compare their training error. The 34-layer plain net has higher training error throughout the whole training procedure, even though the solution space of the 18-layer network is a subspace of that of the 34-layer one. We argue that this optimization difficulty is unlikely to be caused by vanishing gradients. These plain networks are trained with Batch Normalization (BN), which ensures that forward propagated signals have non-zero variances. We also verify that the backward propagated gradients exhibit healthy norms with BN.

Next we evaluate 18-layer and 34-layer residual nets (ResNets). The baseline architectures are the same as the above plain nets, except that a shortcut connection is added to each pair of 3×3 filters. Three major observations emerge. (1) The situation is reversed with residual learning — the 34-layer ResNet is better than the 18-layer ResNet (by 2.8%). More importantly, the 34-layer ResNet exhibits considerably lower training error, suggesting that the degradation problem is well addressed. (2) Compared to its plain counterpart, the 34-layer ResNet reduces the top-1 error by 3.5%, resulting from the successfully reduced training error. (3) The 18-layer plain/residual nets are comparably accurate, but the 18-layer ResNet converges faster at early stages.

Deeper Bottleneck Architectures. Due to concerns on the training time, we modify the building block as a bottleneck design. For each residual function F, we use a stack of 3 layers instead of 2. The three layers are 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×3 layer a bottleneck with smaller input/output dimensions. The 50-layer ResNet is constructed by replacing each 2-layer block with this 3-layer bottleneck block. We further construct 101-layer and 152-layer ResNets by using more bottleneck blocks. Remarkably, although the depth is significantly increased, the 152-layer ResNet (11.3 billion FLOPs) still has lower complexity than VGG-16/19 (15.3/19.6 billion FLOPs).

We conduct more studies on CIFAR-10, focusing on the behaviors of extremely deep networks rather than pushing the state-of-the-art results, using intentionally simple architectures. The plain/residual architectures follow the form with 3×3 convolutions on feature maps of sizes {32, 16, 8}, with filter numbers of {16, 32, 64}. Comparing networks of n = {3, 5, 7, 9}, yielding 20, 32, 44, and 56-layer networks: the deep plain nets suffer from increased depth and exhibit higher training error when going deeper. This phenomenon is similar to that on ImageNet, suggesting that such an optimization difficulty is a fundamental problem.

We further explore an aggressively deep model with n = 200, leading to a 1202-layer network. This network shows no optimization difficulty, achieving a training error < 0.1%. Its test error is still fairly good (7.93%). But there are still open problems on such aggressively deep models: the testing result of this 1202-layer network is worse than that of the 110-layer network, although both have similar training error. We argue that this is because of overfitting. The 1202-layer network may be unnecessarily large (19.4M parameters) for this small dataset. Analysis of Layer Responses shows that ResNets have generally smaller responses than their plain counterparts, supporting the notion that residual functions might be generally closer to zero than the non-residual functions.

Deep residual nets show excellent generalization performance on other recognition tasks. We adopt Faster R-CNN as the detection method and evaluate the improvements solely due to replacing VGG-16 with ResNet-101. Most remarkably, on the challenging COCO dataset, we obtain a 6.0% increase in COCO's standard metric (mAP@[.5, .95]), which is a 28% relative improvement. This gain is solely due to the learned representations. Based on deep residual nets, the authors won 1st places in several tracks of ILSVRC & COCO 2015 competitions: ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.