Recent work has shown that convolutional networks can be substantially deeper, more accurate, and efficient to train if they contain shorter connections between layers close to the input and those close to the output. In this paper, we embrace this observation and introduce the Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion. Whereas traditional convolutional networks with L layers have L connections — one between each layer and its subsequent layer — our network has L(L+1)/2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. DenseNets have several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters.

Convolutional neural networks (CNNs) have become the dominant machine learning approach for visual object recognition. Although they were originally introduced over 20 years ago, improvements in computer hardware and network structure have enabled the training of truly deep CNNs only recently. As CNNs become increasingly deep, a new research problem emerges: as information about the input or gradient passes through many layers, it can vanish and "wash out" by the time it reaches the end (or beginning) of the network. Many recent publications address this or related problems. ResNets, Highway Networks, Stochastic Depth, and FractalNets all share a key characteristic: they create short paths from early layers to later layers.

In this paper, we propose an architecture that distills this insight into a simple connectivity pattern: to ensure maximum information flow between layers in the network, we connect all layers (with matching feature-map sizes) directly with each other. To preserve the feed-forward nature, each layer obtains additional inputs from all preceding layers and passes on its own feature-maps to all subsequent layers. Crucially, in contrast to ResNets, we never combine features through summation before they are passed into a layer; instead, we combine features by concatenating them. Hence, the l-th layer has l inputs, consisting of the feature-maps of all preceding convolutional blocks.

A possibly counter-intuitive effect of this dense connectivity pattern is that it requires fewer parameters than traditional convolutional networks, as there is no need to re-learn redundant feature-maps. Traditional feed-forward architectures can be viewed as algorithms with a state, which is passed on from layer to layer. Each layer reads the state from its preceding layer and writes to the subsequent layer. It changes the state but also passes on information that needs to be preserved. DenseNet layers are very narrow (e.g., 12 filters per layer), adding only a small set of feature-maps to the "collective knowledge" of the network and keeping the remaining feature-maps unchanged — and the final classifier makes a decision based on all feature-maps in the network.

The exploration of network architectures has been a part of neural network research since the early days. The recent resurgence in popularity of neural networks has also revived this interest. Highway Networks were amongst the first architectures that provided a means to effectively train end-to-end networks with more than 100 layers. ResNets have achieved impressive, record-breaking performance on many challenging image recognition, localization, and detection tasks. An orthogonal approach to making networks deeper is to increase the network width. GoogLeNet uses an "Inception module" which concatenates feature-maps produced by filters of different sizes. Stochastic depth shows that not all layers may be needed and highlights that there is a great amount of redundancy in deep (residual) networks. Our paper was partly inspired by that observation.

Consider a single image x_0 that is passed through a convolutional network. The network comprises L layers, each of which implements a non-linear transformation H_l. ResNets add a skip-connection that bypasses the non-linear transformations with an identity function: x_l = H_l(x_{l-1}) + x_{l-1}. An advantage of ResNets is that the gradient can flow directly through the identity function from later layers to earlier layers. However, the identity function and the output of H_l are combined by summation, which may impede the information flow in the network. We propose a different connectivity pattern: we introduce direct connections from any layer to all subsequent layers. Consequently, the l-th layer receives the feature-maps of all preceding layers as input: x_l = H_l([x_0, x_1, ..., x_{l-1}]), where [x_0, x_1, ..., x_{l-1}] refers to the concatenation of the feature-maps produced in layers 0, ..., l-1.

If each function H_l produces k feature-maps, it follows that the l-th layer has k_0 + k * (l-1) input feature-maps, where k_0 is the number of channels in the input layer. An important difference between DenseNet and existing network architectures is that DenseNet can have very narrow layers, e.g., k = 12. We refer to the hyper-parameter k as the growth rate of the network. Each layer adds k feature-maps of its own to this state. The growth rate regulates how much new information each layer contributes to the global state. The global state, once written, can be accessed from everywhere within the network and, unlike in traditional network architectures, there is no need to replicate it from layer to layer.

Although each layer only produces k output feature-maps, it typically has many more inputs. It has been noted that a 1x1 convolution can be introduced as bottleneck layer before each 3x3 convolution to reduce the number of input feature-maps, and thus to improve computational efficiency. We find this design especially effective for DenseNet and we refer to our network with such a bottleneck layer as DenseNet-B. In our experiments, we let each 1x1 convolution produce 4k feature-maps. To further improve model compactness, we can reduce the number of feature-maps at transition layers. If a dense block contains m feature-maps, we let the following transition layer generate floor(theta * m) output feature-maps, where 0 < theta <= 1 is referred to as the compression factor. We refer to our DenseNet with both bottleneck and compression as DenseNet-BC.

We empirically demonstrate DenseNet's effectiveness on several benchmark datasets and compare with state-of-the-art architectures, especially with ResNet and its variants. On CIFAR-10, CIFAR-100, SVHN, and ImageNet, DenseNets obtain significant improvements over the state of the art. DenseNet-BC with L=190 and k=40 outperforms the existing state-of-the-art consistently on all the CIFAR datasets, with error rates of 3.46% on C10+ and 17.18% on C100+, which are significantly lower than the error rates achieved by wide ResNet architecture. On parameter efficiency, DenseNet-BC with L=100 and k=12 achieves comparable performance (e.g., 4.51% vs 4.62% error on C10+, 22.27% vs 22.71% error on C100+) as the 1001-layer pre-activation ResNet using 90% fewer parameters. On ImageNet, a DenseNet-201 with 20M parameters yields similar validation error as a 101-layer ResNet with more than 40M parameters.

We proposed DenseNet, a new convolutional network architecture that introduces direct connections between any two layers with the same feature-map size. We showed that DenseNets scale naturally to hundreds of layers, while exhibiting no optimization difficulties. Feature analysis demonstrates that all layers spread their weights over many inputs within the same block, indicating that features extracted by very early layers are indeed directly used by deep layers throughout the same dense block. This encourages feature reuse throughout the network and leads to more compact models. Because of their compact internal representations and reduced feature redundancy, DenseNets may be good feature extractors for various computer vision tasks that build on convolutional features.