This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we can accelerate training by 3x or more and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods using only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.

Deep learning architectures continue growing in capability and capacity, yet models easily overfit one million images and demand hundreds of millions of labeled images. Self-supervised pre-training has successfully addressed this in natural language processing through autoregressive language modeling (GPT) and masked autoencoding (BERT), enabling training of models exceeding one hundred billion parameters. The methodological gap between NLP and vision motivates a fundamental question: what makes masked autoencoding different between vision and language?

We attempt to answer this question from three perspectives. (i) Architectural differences: Until recently, convolutional networks dominated vision, and convolutions operate on regular grids, making it nontrivial to integrate mask tokens or positional embeddings. Vision Transformers (ViT) have addressed this architectural gap. (ii) Information density: Languages are human-generated signals that are highly semantic and information-dense. Images, in contrast, are natural signals with heavy spatial redundancy — a missing patch can be recovered from neighboring patches with little high-level understanding of parts, objects, and scenes. To overcome this difference, we show that masking a very high portion of random patches (e.g., 75%) largely reduces redundancy and creates a challenging self-supervisory task that requires holistic understanding. (iii) The decoder's role differs between vision and language. In vision, the decoder reconstructs pixels, which is a task of a lower semantic level than recognition. In language, the decoder predicts missing words that contain rich semantic information. In BERT, the decoder can be trivial (an MLP), whereas in our method, the decoder design plays a key role in determining the semantic level of the learned latent representations.

Our MAE approach masks random patches from the input image and reconstructs the missing pixels. It has an asymmetric encoder-decoder design. Our encoder is a ViT that operates only on the visible (unmasked) subset of patches, with no mask tokens. Our decoder is lightweight and reconstructs the full image from the latent representation together with mask tokens. Shifting the mask tokens to the small decoder reduces computation greatly. Under this design, a very high masking ratio (e.g., 75%) achieves a win-win scenario: it optimizes accuracy while allowing the encoder to process only a small portion (e.g., 25%) of patches, reducing overall pre-training time by 3x or more, and similarly reducing memory consumption. This enables training even larger models. MAE can train ViT-Large/-Huge on ImageNet-1K with improved generalization performance. With MAE pre-training, we can train data-hungry models like ViT-Large/-Huge on ImageNet-1K, achieving 87.8% when fine-tuned on ImageNet-1K.

Masked language modeling and its autoregressive counterpart, as represented by BERT and GPT, are highly successful methods for pre-training in NLP. These methods hold out a portion of the input sequence and train models to predict the removed content. They scale excellently and show strong evidence of generalization across downstream tasks. Autoencoding is a classical approach to representation learning: an encoder maps inputs to a latent representation and a decoder reconstructs the input. Denoising autoencoders (DAE), a key variant, corrupt the input signal and learn to reconstruct the original. Masked image modeling can be viewed as a form of denoising autoencoding where the corruption is patch removal.

Masked image encoding methods learn representations from mask-corrupted images. Context Encoder inpaints large missing regions using convolutional networks. More recent Transformer-based methods include iGPT, which operates on pixel sequences, ViT studies that explore masked patch prediction, and BEiT, which predicts discrete visual tokens from a pre-trained tokenizer. Meanwhile, self-supervised contrastive learning methods have gained prominence, modeling image similarity and dissimilarity across augmented views. Contrastive methods depend heavily on data augmentation strategies, whereas autoencoding pursues a conceptually different direction. Our approach is related to all these works but differs in its simplicity and effectiveness: we demonstrate that simple pixel reconstruction with an asymmetric design outperforms or matches more complex token-based or contrastive approaches.

Following ViT, we divide an image into regular non-overlapping patches. Then we sample a subset of patches and mask (i.e., remove) the remaining ones. Our sampling strategy is straightforward: we sample random patches without replacement, following a uniform distribution. We refer to this as "random sampling". Random sampling with a high masking ratio largely eliminates redundancy, thus creating a task that cannot be easily solved by extrapolation from visible neighboring patches. The uniform distribution prevents potential center bias that could arise from other sampling strategies. Furthermore, the highly sparse input creates an opportunity for an efficient encoder design that processes only visible patches.

Our encoder is a Vision Transformer (ViT) but applied only to visible, unmasked patches. Just as in a standard ViT, our encoder embeds patches by a linear projection with added positional embeddings, and then processes the resulting set of tokens through a series of Transformer blocks. However, our encoder only operates on a small subset (e.g., 25%) of the full set. Masked patches are removed; no mask tokens are used. This allows us to train very large encoders with only a fraction of compute and memory. The full set is handled by a lightweight decoder, described next.

The MAE decoder is applied to the full set of tokens consisting of (i) encoded visible patches, and (ii) mask tokens. Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted. We add positional embeddings to all tokens in this full set; without positional embeddings, mask tokens would have no information about their location in the image. The decoder has another series of Transformer blocks. The MAE decoder is only used during pre-training to perform the image reconstruction task. Therefore, the decoder architecture can be flexibly designed in a manner that is independent of the encoder design. We experiment with very small decoders, narrower and shallower than the encoder. For example, our default decoder has <10% computation per token vs. the encoder. With this asymmetrical design, the full set of tokens is only processed by the lightweight decoder, significantly reducing pre-training time.

Our MAE reconstructs the input by predicting the pixel values for each masked patch. Each element in the decoder's output is a vector of pixel values representing a patch. The last layer of the decoder is a linear projection whose number of output channels equals the number of pixel values in a patch. The loss function computes the mean squared error (MSE) between the reconstructed and original images in the pixel space. We compute the loss only on masked patches, similar to BERT. We also study a variant whose reconstruction target is the normalized pixel values of each masked patch. Specifically, we compute the mean and standard deviation of all pixels in a patch and use them to normalize this patch. Using per-patch normalized pixels as the reconstruction target improves representation quality in our experiments.

Simple implementation. MAE pre-training can be implemented efficiently, and does not need any specialized sparse operations. First, we generate tokens for every input patch (with linear projection and positional embeddings). Next, we randomly shuffle the list of tokens and remove the last portion according to the masking ratio. This is equivalent to sampling patches without replacement. After encoding, we append a list of mask tokens to the list of encoded patches, and unshuffle this full list to align all tokens with their targets. The decoder is applied to this full list (with positional embeddings added). As noted, no sparse operations are needed. This simple implementation introduces negligible overhead as random shuffling and unshuffling are fast.

Self-supervised pre-training is performed on ImageNet-1K (IN1K). We then evaluate representations by (i) end-to-end fine-tuning or (ii) linear probing, and report top-1 validation accuracy on ImageNet-1K. We use ViT-Large (ViT-L/16) as the backbone for ablation studies. ViT-L is very large and tends to overfit when trained from scratch: the original ViT-L achieved only 76.5% from scratch, while our improved implementation achieves 82.5% from scratch. MAE pre-training boosts this to 84.9% with fine-tuning for only 50 epochs (vs. 200 from scratch). On masking ratio: the optimal ratio is surprisingly high at 75% for both fine-tuning and linear probing. This is in stark contrast with BERT's typical 15% masking ratio, and also much higher than related work in vision (20-50%).

Decoder design. Our MAE decoder can be flexibly designed. Sufficiently deep decoders are important for linear probing — this is explained by the gap between the pixel reconstruction task and recognition: the last several layers in an autoencoder are more specialized for reconstruction but less relevant for recognition. A reasonably deep decoder can account for the reconstruction specialization, leaving the latent representations at a more abstract level, yielding up to 8% improvement in linear probing. With fine-tuning, however, the last layers of the encoder can be tuned for recognition, and decoder depth is less influential — a single-block decoder performs strongly (84.8%). For decoder width, 512 dimensions perform well; our default decoder has 8 blocks, 512-d width, comprising only 9% of ViT-L FLOPs.

Mask token in encoder. If the encoder uses mask tokens, its accuracy drops by 14% under linear probing. With mask tokens, there is a gap between pre-training and deploying: the encoder has a large portion of mask tokens in its input during pre-training, which do not exist in uncorrupted images during deployment. Removing mask tokens from the encoder constrains it to always see real patches, improving representation quality. Moreover, encoder mask token removal reduces training computation by 3.3x in total FLOPs, enabling a 2.8x wall-clock speedup. Data augmentation. MAE works well with cropping-only augmentation. Surprisingly, it performs decently even without augmentation (center-crop only). This is dramatically different from contrastive learning, where removing augmentation leads to 13% and 28% drops for BYOL and SimCLR respectively. In MAE, random masking plays the role of data augmentation: masks create new training samples at every iteration regardless of augmentation.

Comparisons with previous results. We compare MAE with self-supervised ViT results. For ViT-B, methods perform closely. For ViT-L, gaps between methods become larger, suggesting challenges in reducing overfitting for larger models. MAE scales easily with steady improvements for bigger models. ViT-H (224 size) achieves 86.9% accuracy; fine-tuning at 448 size yields 87.8% using only ImageNet-1K data. Previous best ImageNet-only methods achieved 87.1% (at 512 size) based on advanced network designs. Comparing with BEiT: MAE is more accurate while being simpler and faster. MAE reconstructs pixels rather than discrete tokens; BEiT reported 1.8% degradation when reconstructing pixels with ViT-B. MAE requires no dVAE pre-training and is considerably faster (3.5x per epoch) than BEiT. Even with 1600-epoch training, MAE total pre-training time is 31 hours on 128 TPU-v3 cores, versus MoCo v3's 36 hours for only 300 epochs.

Partial fine-tuning provides insight into representation quality beyond linear probing and full fine-tuning. We fine-tune the last several Transformer blocks while freezing the rest. Fine-tuning only one Transformer block significantly boosts accuracy from 73.5% (linear probing) to 81.0%. Fine-tuning only the final block's MLP sub-block achieves 79.1%, far better than linear probing. Fine-tuning a few blocks (4 or 6) closely approaches full fine-tuning accuracy. Comparisons with MoCo v3 are revealing: MoCo v3 has higher linear probing accuracy but all MAE partial fine-tuning results outperform MoCo v3, with gaps reaching 2.6% when tuning 4 blocks. While MAE representations have lower linear separability, they possess stronger nonlinear features that perform very well when even minimal non-linear heads are tuned. This observation suggests linear separability is not the sole metric for representation quality.

We evaluate transfer learning on object detection and instance segmentation using Mask R-CNN on COCO, with ViT backbones adapted for use with FPN. Compared with supervised pre-training, MAE performs better under all configurations. With ViT-B, MAE exceeds supervised by 2.4 box AP (50.3 vs. 47.9). More significantly, with ViT-L, MAE outperforms supervised by 4.0 box AP (53.3 vs. 49.3). The gains are larger for bigger models, indicating the scalability benefit of MAE pre-training in transfer scenarios. Pixel-based MAE matches or exceeds token-based BEiT, while being simpler and faster. Both MAE and BEiT outperform MoCo v3, which only matches supervised pre-training.

Semantic segmentation on ADE20K using UperNet shows significant improvements over supervised pre-training, e.g., 3.7 mIoU points for ViT-L. Pixel-based MAE outperforms token-based BEiT on this task as well. Classification on other datasets: On iNaturalists, MAE exhibits strong scaling behavior — bigger models improve considerably, surpassing previous best results. On Places, MAE outperforms previous best results that were obtained through pre-training on billions of images. These transfer results collectively demonstrate that MAE pre-training learns representations that generalize well across diverse visual tasks, and the benefits scale with model capacity.

Simple algorithms that scale well are the core of deep learning. In NLP, simple self-supervised pre-training methods (e.g., masked language modeling) have enabled benefits from the exponential scaling of models. In computer vision, supervised pre-training has dominated despite progress in self-supervised methods. This study observes, on ImageNet and in transfer tasks, that autoencoders — a simple self-supervised method conceptually similar to techniques in NLP — provide scalable benefits in computer vision. Self-supervised learning in vision may now follow a similar trajectory as in NLP.

On the other hand, we note that images and languages are signals of a different nature and this difference must be addressed carefully. Images are merely recorded light without a semantic decomposition into the visual analogue of words. Rather than removing objects, our method removes random patches that are unlikely to form a semantic segment. Likewise, our MAE reconstructs pixels, which are not semantic entities. Nonetheless, we observe that our MAE infers complex, holistic reconstructions, suggesting it has learned numerous visual concepts, i.e., semantics. We hypothesize this behavior occurs via the rich hidden representations inside the MAE. We hope this perspective will inspire future work.