While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Self-attention-based architectures, in particular Transformers (Vaswani et al., 2017), have become the model of choice in natural language processing (NLP). The dominant approach is to pre-train on a large text corpus and then fine-tune on a smaller task-specific dataset (Devlin et al., 2019). Thanks to Transformers' computational efficiency and scalability, it has become possible to train models of unprecedented size, with over 100B parameters (Brown et al., 2020; Lepikhin et al., 2020). With the models and datasets growing, there is still no sign of saturating performance.

In computer vision, however, convolutional architectures remain dominant (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016). Inspired by NLP success, multiple works try to combine CNN-like architectures with self-attention (Wang et al., 2018; Carion et al., 2020), some even replacing convolutions entirely (Ramachandran et al., 2019; Wang et al., 2020a). The latter models, while theoretically efficient, have not yet been scaled effectively on modern hardware accelerators due to the use of specialized attention patterns. Therefore, in large-scale image recognition, classic ResNet-like architectures are still state of the art (Mahajan et al., 2018; Xie et al., 2020; Kolesnikov et al., 2020).

Inspired by the Transformer scaling success in NLP, we experiment with applying a standard Transformer directly to images, with the fewest possible modifications. To do so, we split an image into patches and provide the sequence of linear embeddings of these patches as an input to a Transformer. Image patches are treated the same way as tokens (words) in an NLP application. We train the model on image classification in supervised fashion. When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies a few percentage points below comparably sized ResNets. This seemingly discouraging outcome is expected: Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data. However, the picture changes if the models are trained on larger datasets (14M–300M images). We find that large scale training trumps inductive bias. Our Vision Transformer (ViT) attains excellent results when pre-trained at sufficient scale and transferred to benchmarks with fewer datapoints: 88.55% top-1 on ImageNet, 90.72% on ImageNet-ReaL, 94.55% on CIFAR-100, and 77.63% on the VTAB suite of 19 tasks.

Naive application of self-attention to images would require that each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes. Thus, to apply Transformers in the context of image processing, several approximations have been tried in the past. Parmar et al. (2018) applied self-attention only in local neighborhoods for each query pixel instead of globally. Such local multi-head dot-product self-attention blocks can completely replace convolutions (Hu et al., 2019; Ramachandran et al., 2019). In a different line of work, Sparse Transformers (Child et al., 2019) employ scalable approximations to global self-attention. An alternative way to scale attention is to apply it in blocks of varying sizes (Weissenborn et al., 2019), in the extreme case only along individual axes (Ho et al., 2019; Wang et al., 2020a). Many of these specialized attention architectures demonstrate promising results on computer vision tasks, but require complex engineering to be implemented efficiently on hardware accelerators.

Most related to ours is the model of Cordonnier et al. (2020), which extracts patches of size 2x2 from the input image and applies full self-attention on top. This model is very similar to ViT, but our work goes further to demonstrate that large scale pre-training makes vanilla transformers competitive with (or even better than) state-of-the-art CNNs. Moreover, Cordonnier et al. (2020) use a small patch size of 2x2 pixels, which makes the model applicable only to small-resolution images, while we handle medium-resolution images as well. Another recent related model is image GPT (iGPT) (Chen et al., 2020a), which applies Transformers to image pixels after reducing image resolution and color space. The model is trained in an unsupervised fashion as a generative model, and the resulting representation can then be fine-tuned or probed linearly for classification performance, achieving a maximal accuracy of 72% on ImageNet.

To handle 2D images, we reshape the image x ∈ R^H×W×C into a sequence of flattened 2D patches x_p ∈ R^N×(P2·C), where (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each image patch, and N = HW/P² is the resulting number of patches, which also serves as the effective input sequence length for the Transformer. The Transformer uses constant latent vector size D through all of its layers, so we flatten the patches and map to D dimensions with a trainable linear projection. We refer to the output of this projection as the patch embeddings.

Similar to BERT's [class] token, we prepend a learnable embedding to the sequence of embedded patches, whose state at the output of the Transformer encoder serves as the image representation. Both during pre-training and fine-tuning, a classification head is attached to this token. The classification head is implemented by an MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time. Position embeddings are added to the patch embeddings to retain positional information. We use standard learnable 1D position embeddings, since we have not observed significant performance gains from more advanced 2D-aware position embeddings. The Transformer encoder consists of alternating layers of multiheaded self-attention (MSA) and MLP blocks. Layernorm (LN) is applied before every block, and residual connections after every block. The MLP contains two layers with a GELU non-linearity.

We note that Vision Transformer has much less image-specific inductive bias than CNNs. In CNNs, locality, two-dimensional neighborhood structure, and translation equivariance are baked into each layer throughout the whole model. In ViT, only MLP layers are local and translationally equivariant, while the self-attention layers are global. The two-dimensional neighborhood structure is used only very sparingly: in the beginning of the model by cutting the image into patches and at fine-tuning time for adjusting the position embeddings for images of different resolution. Other than that, the position embeddings at initialization time carry no information about the 2D positions of the patches and all spatial relations between the patches have to be learned from scratch.

Typically, we pre-train ViT on large datasets, and fine-tune to (smaller) downstream tasks. For this, we remove the pre-trained prediction head and attach a zero-initialized D×K feedforward layer, where K is the number of downstream classes. It is often beneficial to fine-tune at higher resolution than pre-training. When feeding images of higher resolution, we keep the patch size the same, which results in a larger effective sequence length. The Vision Transformer can handle arbitrary sequence lengths (up to memory constraints), however, the pre-trained position embeddings may no longer be meaningful. We therefore perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image. Note that this resolution adjustment and patch extraction are the only points at which an inductive bias about the 2D structure of the images is manually injected into the Vision Transformer.

We evaluate on three pre-training datasets of increasing size: ILSVRC-2012 ImageNet with 1k classes and 1.3M images; ImageNet-21k with 21k classes and 14M images; and JFT with 18k classes and 303M high-resolution images. We design our Vision Transformer configurations based on those used for BERT: ViT-Base (12 layers, hidden size 768, 12 heads, 86M params), ViT-Large (24 layers, hidden size 1024, 16 heads, 307M params), and ViT-Huge (32 layers, hidden size 1280, 16 heads, 632M params). The brief notation "ViT-L/16" means the "Large" variant with 16×16 input patch size. For our baseline CNNs, we use ResNets (BiT), using Group Normalization instead of Batch Normalization, and standardized convolutions for improved transfer. All models are pre-trained using Adam with β₁=0.9, β₂=0.999, batch size of 4096 and weight decay of 0.1. Fine-tuning uses SGD with momentum, batch size 512.

ViT-H/14 pre-trained on JFT-300M outperforms all prior methods on every benchmark: 88.55% on ImageNet, 90.72% on ImageNet-ReaL, 99.50% on CIFAR-10, 94.55% on CIFAR-100, 97.56% on Oxford-IIIT Pets, 99.68% on Oxford Flowers-102, and 77.63% on the VTAB suite of 19 tasks. The smaller ViT-L/16 pre-trained on JFT-300M outperforms BiT-L (ResNet152x4) on all tasks, while requiring substantially less computational resources to train. The even smaller ViT-L/16 pre-trained on ImageNet-21k performs well on most benchmarks too, while being trainable using a standard cloud TPUv3 with 8 cores in approximately 30 days. In comparison, BiT-L requires 9.9k TPUv3-core-days, and Noisy Student (EfficientNet-L2) requires 12.3k TPUv3-core-days, while ViT-H/14 needs only 2.5k TPUv3-core-days and ViT-L/16 (JFT) only 0.68k.

We explore how the size of the pre-training dataset affects performance. When pre-trained on the smallest dataset, ImageNet, ViT-Large models underperform compared to ViT-Base models, despite (moderate) regularization. With ImageNet-21k pre-training, their performances are similar. Only with JFT-300M do we see the full benefit of larger models. In a second experiment on random subsets of JFT (9M, 30M, 90M, 303M images), Vision Transformers overfit more than ResNets with comparable computational cost on smaller datasets. For example, ViT-B/32 performs much worse than ResNet50 on the 9M subset, but better on 90M+ subsets. This reinforces the intuition that the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial.

We perform a controlled scaling study to evaluate transfer performance from JFT-300M across multiple model families. The study includes 7 ResNets, 6 Vision Transformers, and 5 Hybrids, spanning a wide range of computational budgets. Results show that Vision Transformers dominate ResNets on the performance/compute trade-off. ViT uses approximately 2-4x less compute to attain the same performance (averaged over 5 datasets). Hybrids slightly outperform ViT at small computational budgets, but the difference vanishes for larger models. Importantly, Vision Transformers appear not to saturate within the range tried, motivating future scaling efforts.

To understand how ViT processes image data, we analyze its internal representations. The learned embedding filters of the initial linear projection resemble plausible basis functions for a low-dimensional representation of the fine structure within each patch. The position embeddings encode distance within the image — closer patches tend to have more similar position embeddings. Moreover, the row-column structure appears; patches in the same row/column have similar embeddings. This explains why hand-crafted 2D-aware embedding variants do not yield improvements. For attention distance analysis, some heads attend to most of the image already in the lowest layers, showing that the ability to integrate information globally is indeed used by the model. Other attention heads have consistently small attention distances in the low layers. This mixed behavior is analogous to early convolutional layers in CNNs. The attention distance increases with network depth, and the model learns to attend to image regions that are semantically relevant for classification.

We also explore masked patch prediction for self-supervision, mimicking the masked language modeling task used in BERT. With self-supervised pre-training, the smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, which is a significant improvement of 2% over training from scratch, but still 4% behind supervised pre-training. The self-supervised approach corrupts 50% of patch embeddings and predicts the mean color of the corrupted patches. While preliminary, these results suggest that contrastive pre-training methods such as those explored in the concurrent work of Chen et al. (2020b) may offer further improvements, and exploring self-supervised methods for ViT remains an exciting direction for future work.

We have explored the direct application of Transformers to image recognition. Unlike prior works using self-attention in computer vision, we do not introduce image-specific inductive biases into the architecture apart from the initial patch extraction step. Instead, we interpret an image as a sequence of patches and process it by a standard Transformer encoder as used in NLP. This simple, yet scalable, strategy works surprisingly well when coupled with pre-training on large datasets. Thus, Vision Transformer matches or exceeds the state of the art on many image classification benchmarks, whilst being relatively cheap to pre-train.

While initial results are encouraging, many challenges remain. One is to apply ViT to other computer vision tasks, such as detection and segmentation. Another challenge is to continue exploring self-supervised pre-training methods. Our initial experiment with masked patch prediction shows improvement over training from scratch but still lags behind supervised pre-training. Finally, further scaling of ViT would likely lead to improved performance.