The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing self-supervised methods can produce such features if trained on enough curated data from diverse sources. We train a ViT model with 1 billion parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP, on most benchmarks at image and pixel levels.

Can we learn general-purpose visual features that work across image distributions and tasks without finetuning? In NLP, foundation models pretrained on web-scale text have achieved this goal. In vision, text-guided pretraining (CLIP, OpenCLIP) has shown promise, but we argue it limits the information retained about images — text captions cannot describe all visual details at the pixel level. In contrast, self-supervised learning can capture information at both the image and pixel level, making it a more suitable foundation for general-purpose features. The missing ingredient has been sufficient scale and diversity of training data combined with careful curation.

We construct LVD-142M, a curated dataset of 142 million images. Our pipeline combines curated seed datasets (ImageNet-22k, Google Landmarks, fine-grained classification sets) with images retrieved from 1.2 billion uncurated web images. The retrieval process uses a pretrained ViT-H/16 to compute embeddings, performs k-means clustering, and retrieves nearest neighbors (typically 4 per query image) from the uncurated pool. Copy detection deduplication is applied to remove near-duplicates and increase diversity. This approach ensures the dataset covers diverse visual concepts while maintaining quality through proximity to curated exemplars.

The training combines multiple loss components. The image-level objective (DINO loss) computes cross-entropy between student and teacher network features from the class token using different image crops. The patch-level objective (iBOT loss) randomly masks input patches for the student while keeping them visible for the teacher, applying loss to masked token predictions. Key design modifications at scale include: untying head weights between DINO and iBOT objectives, Sinkhorn-Knopp centering, KoLeo regularizer for uniform feature distribution, and high-resolution adaptation (518x518) in the final training phase.

Training a ViT-g model with 1 billion parameters requires substantial engineering. Key optimizations include: custom FlashAttention implementation for GPU hardware, sequence packing that concatenates variable-length token sequences with block-diagonal attention masks, efficient stochastic depth that skips computation of dropped residuals, and FSDP (Fully Sharded Data Parallelism) for distributed training across GPUs with mixed-precision. An important additional contribution is model distillation: smaller models distilled from the ViT-g outperform models of the same size trained from scratch on all 12 benchmarks.

DINOv2 ViT-g/14 achieves 86.5% linear evaluation accuracy on ImageNet-1k, surpassing previous self-supervised methods by +4.2% over iBOT and outperforming OpenCLIP-G (+0.3%) and EVA-CLIP (+0.1%). Domain robustness is dramatically improved: ImageNet-A: +29.6% over iBOT; ImageNet-R: +22.1%; Sketch: +23.0%. For dense recognition tasks: ADE20k semantic segmentation reaches 49.0 mIoU with a linear probe; depth estimation on NYUd achieves 0.344 RMSE with a single linear layer. These frozen-backbone results approach or match fully fine-tuned MAE (53.6 mIoU on ADE20k), demonstrating the features' out-of-the-box utility.

Ablation studies reveal the contribution of each component. The KoLeo regularizer provides +2.3% k-NN improvement and +8% retrieval boost. Masked image modeling (iBOT loss) is critical for dense predictions (+3% on segmentation). LVD-142M outperforms uncurated data while maintaining ImageNet performance, confirming that data curation — not just data scale — is essential for learning universal features. Model and data scaling curves show that larger models increasingly benefit from the larger, more diverse LVD-142M dataset compared to ImageNet-22k.

DINOv2 demonstrates that self-supervised pretraining on curated, diverse data can produce visual features that match or exceed text-guided approaches across a wide range of tasks. The success rests on four pillars: improved training recipes, larger model scale, curated LVD-142M dataset, and knowledge distillation. The resulting features exhibit emergent properties including automatic foreground-background separation, semantic part alignment across categories, and strong cross-domain transfer. We believe this work demonstrates that self-supervised learning is a viable — and potentially superior — path to general-purpose visual features.