We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective of contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks, outperforming its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC and COCO. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

In natural language processing (NLP), unsupervised representation learning has been enormously successful, as exemplified by GPT and BERT. In computer vision, however, supervised pre-training still dominates. The reason may be related to their different signal spaces: language tasks use discrete signal spaces (words, sub-word units) that are amenable to building tokenized dictionaries, while vision tasks use continuous, high-dimensional signals that are not structured for dictionary building.

We hypothesize that it is possible to build an effective visual dictionary for contrastive learning. We present Momentum Contrast (MoCo) as a way of building large and consistent dictionaries for unsupervised learning with a contrastive loss. We maintain the dictionary as a queue of data samples: the encoded representations of the current mini-batch are enqueued, and the oldest are dequeued. The dictionary keys are encoded by a slowly progressing (momentum-updated) encoder, making the key representations more consistent across different mini-batches.

Contrastive learning traces back to contrastive losses and NCE (Noise-Contrastive Estimation). Recent work includes InstDisc which stores features in a memory bank for all images, and SimCLR which relies on large batch sizes for a large set of negatives. The memory bank approach suffers from stale representations since stored features are from different training epochs. The large-batch approach requires expensive multi-GPU synchronization and is limited by GPU memory. MoCo offers a third perspective that decouples dictionary size from mini-batch size.

We consider contrastive learning as training an encoder for a dictionary look-up task. Consider an encoded query q and a set of encoded samples {k_0, k_1, k_2, ...} that are the keys of a dictionary. Assume there is a single key k_+ that q matches. A contrastive loss (InfoNCE) is a function whose value is low when q is similar to its positive key k_+ and dissimilar to all other keys. The loss takes the form of a softmax-based classifier that classifies q as k_+ among K negative keys.

The dictionary as a queue allows us to decouple the dictionary size from the mini-batch size. The dictionary size can be much larger than a typical mini-batch size, and can be flexibly and independently set as a hyperparameter. The samples in the dictionary are progressively replaced: the current mini-batch is enqueued and the oldest mini-batch is dequeued. The momentum update of the key encoder uses the formula theta_k = m * theta_k + (1-m) * theta_q, where m = 0.999 works much better than smaller values like 0.9. This slow update makes the key encoder evolve smoothly, so the representations in the queue remain relatively consistent.

Under the linear classification protocol on ImageNet, MoCo with a ResNet-50 encoder achieves 60.6% top-1 accuracy, competitive with SimCLR's 69.3% (which uses 8x larger batch size and stronger augmentation). The key finding is in transfer learning: when fine-tuned on PASCAL VOC object detection, MoCo pre-training achieves AP of 57.4 vs. 57.2 for supervised ImageNet pre-training. On COCO object detection and instance segmentation, MoCo outperforms the supervised counterpart in all 7 metrics. These results demonstrate that unsupervised pre-training can surpass supervised pre-training when the target task is different from ImageNet classification.

Ablation experiments reveal several important findings. Dictionary size matters: increasing K from 256 to 65536 steadily improves performance. The momentum coefficient m = 0.999 significantly outperforms m = 0.9 (60.6% vs. 55.2% on ImageNet linear evaluation), confirming that representation consistency is crucial for contrastive learning. Compared to the end-to-end approach (limited by batch size) and the memory bank approach (inconsistent features), MoCo achieves the best trade-off between dictionary size and consistency.

Our experiments show that MoCo is an effective mechanism for building large and consistent dictionaries for unsupervised contrastive learning. The learned representations transfer strongly to downstream detection and segmentation tasks, in many cases surpassing their supervised pre-training counterparts. We believe MoCo is a simple and general framework applicable beyond visual representation learning. We hope this work will help rethink the role of supervised pre-training in computer vision.