Much of the recent progress made in image classification research can be credited to training procedure refinements, such as changes in data augmentations and optimization methods. The authors examine a collection of training refinements and evaluate their impact through ablation studies. They raise ResNet-50's top-1 validation accuracy from 75.3% to 79.29% on ImageNet. The improvements also translate to better transfer learning performance in object detection and semantic segmentation.

Deep convolutional neural networks (CNNs) have dominated image classification since AlexNet in 2012. Various architectures emerged — VGG, NiN, Inception, ResNet, DenseNet, NASNet. "These advancements did not solely come from improved model architecture. Training procedure refinements, including changes in loss functions, data preprocessing, and optimization methods also played a major role." The authors examine "a collection of training procedure and model architecture refinements that improve model accuracy but barely change computational complexity." Their ResNet-50 with all tricks outperforms SE-ResNeXt-50 while maintaining similar computational costs.

The baseline training procedure includes: random crop with aspect ratio in [3/4, 4/3], horizontal flip with 0.5 probability, PCA noise addition, and RGB normalization. Model parameters use Xavier initialization. Training uses Nesterov SGD for 120 epochs with batch size 256, initial learning rate 0.1 divided by 10 at epochs 30, 60, 90. This baseline achieves 75.87% top-1 accuracy.

Four heuristics are proposed to make large-batch training effective: Linear Scaling Learning Rate — scaling the initial learning rate as "0.1 × b/256" where b is the new batch size; Learning Rate Warmup — because "at the beginning all parameters are far from the final solution", a gradual ramp-up prevents training instability; Zero γ — initializing the batch normalization γ parameter to 0 at residual block endpoints, effectively turning residual blocks into identity mappings initially; and No Bias Decay — applying weight decay only on convolutional and fully-connected layer weights, not biases or BN parameters.

Modern hardware like the Nvidia V100 offers dramatically different throughput at different precisions: "14 TFLOPS in FP32 but over 100 TFLOPS in FP16." By adopting mixed-precision training (FP16) with a batch size of 1024, per-epoch training time is reduced from 13.3 to 4.4 minutes while accuracy improves by 0.5%.

Three incremental modifications to the ResNet architecture are proposed: ResNet-B — swap stride positions in the downsampling residual blocks to preserve information flow through the 1×1 convolution; ResNet-C — replace the initial 7×7 convolution "with three consecutive 3×3 convolutions", maintaining the same receptive field while reducing parameters; ResNet-D — add "a 2×2 average pooling layer with stride 2 before the convolution" in the shortcut connection of downsampling blocks. The combined tweaks yield approximately 1% improvement in top-1 accuracy, with practical throughput decrease of only 3%.

Cosine learning rate decay "potentially improves the training progress" compared to step decay. Instead of abruptly reducing the learning rate at fixed epochs, the cosine schedule smoothly decreases it following a half-cosine curve, providing a more gradual transition that allows the model to continue learning effectively throughout training.

Label smoothing replaces the hard one-hot target with a smoothed distribution: "1−ε for the correct class, ε/(K−1) for others", where ε is a small constant and K is the number of classes. This approach encourages "finite output and can generalize better" by preventing the model from becoming over-confident on training examples.

Knowledge distillation "uses a teacher model to help train the current model (student)." The student network learns not only from the hard labels but also from the soft probability distribution produced by a pre-trained, typically larger teacher model. This transfers the teacher's learned knowledge of inter-class similarities to the student.

Mixup training "randomly samples two examples and forms a new example by weighted linear interpolation" of both the inputs and their labels. This data augmentation technique effectively regularizes the model by encouraging linear behavior between training examples in the input space.

Stacking all refinements yields cumulative gains: cosine decay contributes +0.75%, label smoothing adds +0.4%, and mixup contributes +0.84%, ultimately achieving 79.29% top-1 accuracy — a total gain of nearly 4 percentage points over the original baseline.

The trained models are evaluated on downstream tasks. For Faster-RCNN on PASCAL VOC, the best classifier achieves 81.33% mAP. For FCN on ADE20K, cosine decay proves effective, but "models trained with label smoothing, distillation and mixup favor softened labels, blurred pixel-level information may degrade overall pixel-level accuracy." This reveals a nuanced trade-off: techniques that improve classification may not universally benefit dense prediction tasks.

The paper presents twelve techniques for improving CNN training, introducing "minor modifications to the model architecture, data preprocessing, loss function, and learning rate schedule." While individually modest, collectively they significantly boost accuracy and benefit transfer learning across multiple downstream tasks.