We launch EVA, a vision-centric foundation model to "explore the limits of visual representation at scale using only publicly accessible data." EVA is a vanilla ViT pre-trained to reconstruct masked image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new state-of-the-art performance on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation, and semantic segmentation without heavy supervised training.

We observe "quantitative changes in scaling EVA result in qualitative changes in transfer learning performance," especially in large vocabulary instance segmentation. EVA achieves comparable performance on LVISv1.0 with over 1,200 categories and COCO with only 80 categories, nearly closing the gap between these two benchmarks. Besides, we also find that EVA, as a vision-centric multi-modal pivot, can connect images and text. Using EVA to initialize the vision tower of a giant CLIP model significantly improves training efficiency and stability, achieving 78.5% zero-shot top-1 accuracy on ImageNet-1K with fewer samples and less compute.

"Scaling up pre-trained language models has revolutionized natural language processing." The key to this success lies in the simple and scalable self-supervised learning task of masked signal prediction, which enables Transformers to scale to billions of parameters and achieve remarkable performance across a wide range of downstream tasks. Inspired by this paradigm, the vision community has made considerable efforts to replicate this success. However, "the most competitive billion-sized vision pre-trained models still heavily rely on supervised or weakly-supervised training with hundreds of millions of labeled data," presenting a stark contrast with the self-supervised scaling trend in NLP.

The authors attribute this gap partly to the nature of visual information: "natural images are raw and information-sparse." They argue that "an ideal vision pretext task needs the abstraction of not only low-level geometry and structure information, but also high-level semantics, which is hardly captured by pixel-level recovery tasks" such as MAE. While masked image modeling (MIM) has shown promising results at smaller scales, simply recovering raw pixels or low-level features does not provide sufficient semantic abstraction to fully exploit the scaling potential of vision Transformers.

Through pilot studies, the authors found that "simply using image-text aligned (i.e., CLIP) vision features as the prediction targets in MIM scales up well." This pretext task combines "benefits from both the high-level semantic abstraction of image-text contrastive learning as well as the good capture of geometry and structure in masked image modeling." Based on this finding, they present EVA, a one billion parameter vanilla Vision Transformer pre-trained on 29.6 million publicly accessible images. EVA achieves state-of-the-art results across image classification, video action recognition, object detection, instance segmentation, and semantic segmentation.

Masked Image Modeling (MIM) has emerged as a powerful self-supervised learning paradigm for vision. Early works such as ViT and iGPT reported the "first meaningful MIM pre-training results." The BEiT family significantly improved performance "via masked visual token prediction," using a discrete visual tokenizer to convert image patches into tokens and then predicting masked tokens. More recent approaches such as MAE and SimMIM explore pixel or feature regression in MIM, demonstrating that even simple pixel-level reconstruction can learn useful representations, but only at relatively small model and data scales.

Vision Foundation Models have evolved significantly over recent years. Convolutional neural networks (ConvNets) "have long been the de-facto standard" for visual representation learning. However, "at sufficient model and data scales, ConvNets lag behind ViTs due to lack of scalable pre-training tasks." Large pre-trained ViTs such as SwinV2-G with hierarchical architectures and BEiT-3 with multi-modal representations have pushed the boundaries, but often require custom architectural modifications or massive supervised datasets. EVA demonstrates that "vanilla ViT can be efficiently scaled up to billion-scale parameters" through the right self-supervised pretext task, without the need for hierarchical designs or supervised pre-training.

Contrastive Language-Image Pre-training (CLIP) has demonstrated that aligning visual and textual representations through large-scale contrastive learning yields highly transferable visual features. CLIP and its variants learn joint image-text embeddings from hundreds of millions of image-text pairs, producing vision encoders with remarkable zero-shot transfer capabilities. Works such as MVP and MILAN have explored using CLIP features as prediction targets in masked image modeling, but have not demonstrated the scalability of this approach to billion-parameter models. EVA builds upon these insights and extends the paradigm to a significantly larger scale, revealing emergent properties in transfer learning.

The team first evaluated two candidate approaches for the MIM pretext task: recovering masked tokenized semantic vision features (quantizing CLIP features into discrete tokens) and feature distillation (direct regression to CLIP features). Pilot experiments revealed critical insights: "the (additional) CLIP feature tokenization process is unnecessary for achieving good downstream performance" and "feature distillation fails to provide consistent performance gain as the pre-training becomes longer." Based on these findings, they selected the simplest approach: "simply reconstructing the masked out CLIP vision features conditioned on visible image patches" as the final pretext task.

The authors are candid about the origins of their approach: "this MIM pretext task is not originally proposed by us," crediting MVP and MILAN as prior works that first explored CLIP features as MIM targets. However, they emphasize that "this work shows that this pretext task can scale up to billion-scale parameters and tens of millions of unlabeled images" without requiring semantic feature quantization or image-text paired pre-training data. The contribution is thus positioned as a scaling validation rather than a methodological invention, complemented by comprehensive empirical evaluation across diverse tasks.

EVA adopts a vanilla Vision Transformer (ViT) architecture with 1.0 billion parameters. The configuration comprises 40 Transformer layers, a hidden dimension of 1408, an MLP dimension of 6144, and 16 attention heads, processing images with a 14x14 patch size. Notably, EVA does not employ any hierarchical design, windowed attention, or other architectural modifications commonly found in recent vision foundation models. This "vanilla" design choice is deliberate — it aims to demonstrate that the right pretext task, rather than complex architectural engineering, is the key enabler for scaling visual representation learning.

The training objective is to reconstruct masked image-text aligned vision features from visible patches. EVA employs "block-wise masking with a masking ratio of 40%," corrupting a portion of input patches with [MASK] tokens. The reconstruction targets come from the "publicly available OpenAI CLIP-L/14 vision tower trained on 224x224 pixel images." The output feature of EVA is "first normalized and then projected to the same dimension as the CLIP feature via a linear layer," and the training loss is "negative cosine similarity" between the predicted and target features. Only the features at masked positions contribute to the loss.

Pre-training leverages 29.6 million publicly accessible images aggregated from ImageNet-21K, CC12M, CC3M, Object365, COCO, and ADE20K. The authors note that the CLIP features used as targets "draw benefits from a 400 million image-text dataset" but argue that CLIP is "widely used in other state-of-the-art representation learning and pre-training works" as well. Training employs AdamW optimization with 0.05 weight decay, a peak learning rate of 1e-3 with cosine decay, and runs for 150 epochs with batch size 4096 on 224x224 resolution images. Regularization includes stochastic depth with a rate of 0.1 and RandResizeCrop (0.2, 1.0) for data augmentation. The entire pre-training is completed on 128 NVIDIA A100 40GB GPUs using DeepSpeed ZeRO stage-1 optimization in fp16 precision, finishing in approximately 14.5 days.

On ImageNet-1K, EVA achieves 89.6% top-1 accuracy with 336x336 inputs, further improving to 89.7% at higher resolution. The training pipeline includes intermediate fine-tuning on ImageNet-21K for 60 epochs at 224x224 resolution, followed by 10 epochs on ImageNet-1K. Notably, "EVA simply uses a linear layer as the classifier," contrasting with competitors that rely on "multi-head attention pooling and additional pre-trained language towers." On robustness evaluation across ImageNet-V2, ReaL, Adversarial, Rendition, and Sketch variants, EVA demonstrates "the highest averaged accuracy" with "the smallest performance gap" of only 5.6 percentage points between original ImageNet and variant benchmarks.

For video action recognition, EVA achieves top-1 accuracies of 89.7% on Kinetics-400, 89.8% on Kinetics-600, and 82.9% on Kinetics-700. The approach uses "spatial-temporal attention with no specific architectural adaptation for video." Training involves two stages: intermediate fine-tuning on a merged Kinetics-722 dataset (0.63 million videos, 722 classes) for 40 epochs, then fine-tuning on individual datasets for 1-2 epochs. Testing employs "multi-view inference with 4 temporal clips and 3 spatial crops." These results demonstrate that EVA's visual representations generalize effectively from images to video without requiring video-specific pre-training.

On COCO, EVA achieves 64.7 AP^box and 55.5 AP^mask on test-dev with test-time augmentation. The paper highlights that LVIS presents a "much harder benchmark than COCO" with "more than 1,200 object categories" featuring a long-tail distribution. Remarkably, EVA achieves "55.0 AP^mask on both LVIS val and COCO val," effectively closing the performance gap that conventional methods exhibit between these two benchmarks. While the authors acknowledge "it is inaccurate to say EVA 'solves' the LVIS large vocabulary instance segmentation task," they argue that the achievement of zero gap in AP^mask represents a "significant breakthrough" reflecting "quantitative changes in scaling" producing "qualitative changes in transfer learning performance."

For semantic segmentation, EVA achieves 62.3 mIoU^ms on ADE20K with multi-scale evaluation and 53.4 mIoU^ss on COCO-Stuff with single-scale evaluation. The approach follows "ViT-Adapter with Mask2Former as the segmentation head" but uses "weakened architectural configurations due to GPU memory limitations." Despite these constraints, EVA establishes new state-of-the-art results, demonstrating that strong pre-trained representations can compensate for reduced architectural complexity in downstream tasks. The consistent improvements across segmentation benchmarks reinforce the universal representational quality of EVA's pre-training.

EVA serves as the vision tower initialization for a 1.1 billion parameter CLIP model (EVA CLIP), trained on LAION-400M with "11 billion samples seen" — compared to competitors using 12-32 billion samples. The resulting model achieves "78.5% zero-shot top-1 accuracy on ImageNet-1K without using any training set labels." EVA CLIP outperforms Open CLIP-H and Open CLIP-g on 10 of 12 zero-shot classification benchmarks while using "3x fewer GPUs" and a "~5x smaller dataset."

A key practical advantage is training stability. The authors report that "using fp16 format with dynamic loss scaling is stable enough" for EVA CLIP training, while competitors require the more expensive bfloat16 format to avoid divergence. This stability advantage is attributed to EVA's pre-trained initialization providing a better starting point in the optimization landscape. Overall, EVA CLIP achieves "new state-of-the-art results among all existing self-supervised learning methods" for zero-shot, linear probing, and fine-tuning evaluations on ImageNet-1K, establishing EVA as a versatile foundation for both vision-only and vision-language tasks.

The authors conclude that they "launch EVA, a one billion parameters vanilla ViT encoder to explore the limits of masked visual representation learning." They demonstrate that "simple masked feature modeling as a visual learning pretext task scales well on an architecture with minimal vision priors," achieving state-of-the-art results across image recognition, video action recognition, object detection, instance segmentation, and semantic segmentation — all from a single pre-trained representation. The key insight is that combining the semantic richness of CLIP features with the structural learning of masked image modeling unlocks the scaling potential that pixel-level reconstruction alone could not achieve.

Looking forward, the authors express their hope that "EVA would bridge the gap between vision and language study via masked modeling, and contributes to the Neon Genesis of vision research." EVA's role as a vision-centric multi-modal pivot — connecting images and text through its CLIP-aligned representations — suggests a promising direction where strong visual pre-training serves as the foundation for increasingly capable multi-modal systems. The demonstrated efficiency gains in CLIP training further indicate that the "pre-train then align" paradigm may be more sample-efficient and stable than training multi-modal models from scratch.