We report competitive results on object detection and instance segmentation on the COCO dataset using standard models trained from random initialization. The results are no worse than their ImageNet pre-training counterparts even when using the hyper-parameters of the baseline system (Mask R-CNN) that were optimized for fine-tuning pre-trained models, with the sole exception of increasing the number of training iterations so the randomly initialized models may converge. Training from random initialization is surprisingly robust: the models are no worse than their pre-training counterparts under multiple settings including training with limited data, using deeper and wider models, and performing multiple tasks. We achieved 50.9 AP on COCO object detection without using any external data — a result on par with the top COCO 2017 competition results.

Deep convolutional neural networks have revolutionized computer vision through transfer learning, where representations learned on pre-training tasks transfer to target tasks. The established paradigm has been to pre-train on large-scale data like ImageNet, then fine-tune on target tasks with less training data. This approach enabled state-of-the-art results on object detection, image segmentation, and action recognition. Recent efforts have pushed this paradigm further by pre-training on datasets up to 3000x larger than ImageNet. However, improvements on object detection in particular are small and scale poorly with the pre-training dataset size.

We question this paradigm by exploring the opposite direction: achieving competitive object detection and instance segmentation accuracy when training on COCO from random initialization ("from scratch"), without any pre-training. Notably, this is accomplished using baseline systems and their hyper-parameters that were optimized for fine-tuning pre-trained models. Success requires two key ingredients: appropriate normalization techniques for optimization and sufficient training duration to compensate for lack of pre-training.

Our key observations are threefold. (i) ImageNet pre-training accelerates early convergence but does not prevent models trained from scratch from eventually matching performance after comparable total computation. (ii) Pre-training does not automatically provide regularization; models trained from scratch match accuracy without any extra regularization, even with only 10% COCO data. (iii) Pre-training shows minimal benefit for tasks requiring spatially localized predictions, with noticeable AP improvement for high box overlap thresholds when training from scratch.

Normalization is critical for training object detectors from scratch. Batch Normalization (BN), standard in modern networks, creates difficulties because object detectors are typically trained with high resolution inputs, which reduces batch sizes as constrained by memory, and small batch sizes severely degrade BN accuracy. We address this with two strategies: Group Normalization (GN), which performs computation independent of the batch dimension and is insensitive to batch sizes; and Synchronized Batch Normalization (SyncBN), which computes batch statistics across multiple GPUs to increase the effective batch size.

Training duration requires careful consideration. Pre-training involves over one million images iterated for one hundred epochs, which teaches both high-level semantics and low-level features (edges, textures) that don't require re-learning during fine-tuning. Models trained from scratch must learn both low- and high-level semantics, so more iterations may be necessary. However, a fairer comparison accounts for the total number of training samples (images, instances, pixels) seen across all training stages. When measured in pixels, from-scratch models see comparable samples due to detectors' higher-resolution inputs.

We also report that using appropriately normalized initialization, we can train object detectors with VGG nets from random initialization without BN or GN. This demonstrates that the methodology generalizes beyond modern architectures with normalization layers, suggesting that the difficulty of training detectors from scratch has been largely an optimization issue rather than a fundamental limitation.

On COCO object detection and instance segmentation using Mask R-CNN with FPN and GN, models trained from random initialization match their pre-training counterparts given sufficient training. With a 6x schedule, ResNet-50 trained from scratch achieves 41.3 AP^bbox (vs. 41.1 pre-trained) and 36.6 AP^mask (vs. 36.4 pre-trained). ResNet-101 achieves 42.7 AP^bbox (vs. 42.2 pre-trained) and 37.6 AP^mask (vs. 37.2 pre-trained). Notably, from-scratch models show stronger performance at high overlap thresholds: AP75^bbox is 45.6 vs. 44.6 for R50, confirming that pre-training does not help spatial localization.

For large models trained from scratch, a ResNeXt-152 (8x32d) with approximately 4x FLOPs compared to R101 achieves remarkable results. With training-time augmentation and Cascade R-CNN, the model reaches 48.6 AP^bbox and 41.4 AP^mask. Adding test-time augmentation, it achieves 50.9 AP^bbox and 43.2 AP^mask on val2017, and 51.3 AP^bbox and 43.6 AP^mask on the test-challenge set — matching COCO 2017 competition winners. Remarkably, the same model with ImageNet pre-training achieved only 50.3/42.5 on val2017, showing pre-training provides no advantage for large models.

We investigate the data regime where pre-training might help. With 35k COCO images (~1/3), after grid search optimization, both pre-trained and from-scratch models achieve 36.3 AP. With 10k COCO images (~1/10), the from-scratch model achieves 25.9 AP vs. pre-training's 26.0 AP — still comparable. However, with only 1k images (~1/100), pre-training achieves 9.9 AP vs. 3.5 AP from scratch, revealing the breakdown point between 3.5k and 10k COCO training images. On PASCAL VOC (15k images, 20 categories), pre-training retains an advantage: 82.7 mAP vs. 77.6 mAP from scratch, attributed to VOC's lower instance density.

Our findings challenge the dominant ImageNet pre-training paradigm with six key conclusions. Training from scratch on target tasks is possible without architectural changes. It requires more iterations to converge, but can be no worse than pre-training counterparts under many circumstances, down to as few as 10k COCO images. ImageNet pre-training speeds up convergence but does not necessarily provide regularization unless in very small data regimes. Pre-training helps less if the target task is more sensitive to localization than classification.

We conclude that ImageNet pre-training is a historical workaround for when the community did not have enough target data or computational resources. Collecting data and training on target tasks is a solution worth considering, especially when there is a significant gap between the source pre-training task and the target task. The community should be more careful when evaluating pre-trained features (e.g., for self-supervised learning), as now we learn that even random initialization could produce excellent results.