State of the art visual perception models for a broad range of tasks rely on supervised pretraining. ImageNet-1k has enabled much of the progress to date. However, the question of whether these models can benefit from even more data remains open. In this work, we present a study of transfer learning with large scale weakly-supervised pretraining. We use up to 3.5 billion images from Instagram with noisy hashtag labels and train models on this data, exploring transfer to a variety of vision tasks.

Our experiments show that weakly supervised pretraining is a viable strategy that can surpass ImageNet pretraining on multiple downstream tasks. We observe consistent improvements on image classification, object detection, and instance segmentation when scaling up the amount of pretraining data. With a ResNeXt-101 32x48d model pretrained on 940 million images with 1.5k hashtag labels, we achieve 85.4% top-1 accuracy on ImageNet, which is a new state of the art.

The supervised pretraining paradigm on ImageNet has been remarkably successful across computer vision. However, ImageNet contains only about 1.28 million training images spanning 1,000 classes, which may be a limiting factor as model capacity continues to grow. Can we do better by training on vastly larger datasets, even if the labels are noisy? This is the central question we investigate.

We leverage publicly available images from Instagram where users annotate their photos with hashtags. While these hashtags are noisy and not curated for visual recognition, they provide a scalable and essentially free source of supervision. We train ResNeXt models of varying capacity on datasets of up to 3.5 billion images and systematically study the effect of data size, label noise, model capacity, and transfer learning methodology on downstream task performance.

Our approach consists of three stages. First, we collect images from Instagram and map hashtags to a predefined label vocabulary. We experiment with different vocabulary sizes: 1.5k, 8.5k, and 17k labels. Second, we pretrain ResNeXt models on this weakly-labeled dataset using standard cross-entropy loss. We explore models ranging from ResNeXt-50 to ResNeXt-101 32x48d with varying width multipliers. Third, we fine-tune the pretrained models on target datasets for downstream tasks including image classification, object detection, and instance segmentation.

A critical aspect of our approach is the hashtag-to-label mapping. Instagram hashtags are free-form text and can be highly noisy. We use a simple mapping strategy that maps each hashtag to the closest WordNet synset in the target vocabulary. For multi-label images, we apply a softmax over all matched labels. Despite the simplicity of this approach, we find that the noise in the labels does not prevent the models from learning useful visual representations, as long as the dataset is sufficiently large.

We present results on multiple downstream tasks. For ImageNet classification, our best model achieves 85.4% top-1 accuracy, using a ResNeXt-101 32x48d pretrained on 940M images with 1.5k hashtag labels. This represents a significant improvement over the 82.7% achieved by the same architecture pretrained on ImageNet-1k alone. We observe a clear log-linear relationship between the amount of pretraining data and transfer accuracy, suggesting that further gains could be achieved with even more data.

For object detection and instance segmentation on COCO, weakly supervised pretraining consistently improves over ImageNet pretraining. Using Mask R-CNN with Feature Pyramid Network (FPN), our pretrained models achieve improvements of 1-2 AP points across different model sizes. Importantly, the improvements are more pronounced for larger models, suggesting that bigger models benefit more from larger pretraining datasets, and the capacity of current models may be a bottleneck.

We also investigate the effect of label vocabulary size and find that larger vocabularies generally lead to better transfer performance, but with diminishing returns beyond 8.5k labels. Additionally, we study the interaction between model capacity and data size: small models saturate quickly with increasing data, while large models continue to improve. This suggests that the common practice of using small models to validate ideas on small datasets may be misleading, as the trends may not extrapolate to larger scales.

We have presented a comprehensive study of transfer learning with large-scale weakly supervised pretraining. Our results demonstrate that training on billions of weakly labeled images significantly improves visual representations for a wide range of downstream tasks. The improvements are consistent and follow a log-linear scaling behavior. We hope our work encourages the community to rethink the role of data scale in visual representation learning and to explore more scalable approaches to supervision.