Large-scale NLP models have been shown to significantly improve the performance on language tasks with no signs of saturation. They also demonstrate amazing few-shot capabilities similar to that of human beings. This paper aims to explore large-scale models in computer vision. The authors address three major issues in training and application of large vision models, including training instability, resolution gaps between pre-training and fine-tuning, and hunger for labelled data.

Three main techniques are proposed: (1) a residual-post-norm method combined with cosine attention to improve training stability; (2) a log-spaced continuous position bias method that can effectively transfer models pre-trained using low-resolution images to downstream tasks with high-resolution inputs; (3) a self-supervised pre-training method, SimMIM, to reduce the needs of vast labeled images. Using these techniques, the authors successfully trained a 3 billion-parameter Swin Transformer V2 model, the largest dense vision model to date, with images of up to 1,536 x 1,536 resolution. It set new performance records on 4 representative vision benchmarks: ImageNet-V2 image classification (84.0%), COCO object detection (63.1/54.4 box/mask AP), ADE20K semantic segmentation (59.9 mIoU), and Kinetics-400 video classification (86.8%). Notably, the training required 40x less labelled data and 40x less training time compared with the previous largest vision model.

The scaling up of language models has been one of the most impactful advances in AI. Since the BERT large model with 340 million parameters, language models have been quickly scaled up by more than 1,000 times in a few years, reaching 530 billion dense parameters and 1.6 trillion sparse parameters. With increased capacity, the accuracy of various language benchmarks has been significantly improved, and more importantly, these models demonstrate amazing few-shot capabilities similar to that of human beings, which is considered a critical step toward artificial general intelligence.

In comparison, the scaling up of vision models has lagged behind significantly. While larger vision models generally perform better, the absolute model size was just able to reach about 1-2 billion parameters very recently. More critically, existing large vision models are applied to the image classification task only. Previous approaches such as ViT-G and CoAtNet rely on a huge image dataset with classification labels (i.e., JFT-3B) and have not demonstrated effectiveness on general vision tasks such as object detection and semantic segmentation, which are of critical importance for real-world visual understanding.

To successfully train large general-purpose vision models, the authors identify three key issues. First, training instability: experiments reveal that the discrepancy of activation amplitudes across layers becomes significantly greater in large models. In the pre-normalization configuration, residual unit outputs directly accumulate, causing activation values at deeper layers to be significantly larger than those at early layers — with discrepancies reaching an extreme value of 10^4. Second, resolution gaps: downstream tasks often require high-resolution inputs or large attention windows, and bi-cubic interpolation of position bias maps is ad-hoc and sub-optimal. Third, data hunger: larger models demand more data, and previous approaches require massive labeled datasets like JFT-3B.

Transformer has served as the standard network architecture since the pioneer work in NLP. The exploration of scaling this architecture has since begun, and the progress has been accelerated by the invention of effective self-supervised learning approaches, such as masked or auto-regressive language modeling. The capacity increase has been dramatic: from BERT-340M to the Megatron-Turing-530B and sparse Switch-Transformer-1.6T. With increased capacity, the accuracy of various language benchmarks has been significantly improved, and zero-shot or few-shot performance is also significantly improved, which is regarded as foundational to human-like intelligence.

CNNs have long been the standard computer vision networks. From AlexNet through ResNet, architectures became progressively deeper and larger, advancing visual tasks. However, CNN architectures have been further scaled up to about 1 billion parameters, but absolute performance may not be so encouraging, perhaps due to inductive biases in the CNN architecture limiting modeling power. Recently, Vision Transformers started taking over one representative visual benchmark after another, including ImageNet-1K, COCO object detection, ADE20K semantic segmentation, and Kinetics-400 action classification. Despite numerous variants, only a few works have attempted to scale up vision Transformers, and these rely on a huge image dataset with classification labels (JFT-3B) and are only applied to image classification problems.

For transferring across window or kernel resolutions, CNN kernel sizes typically remain fixed during pre-training and fine-tuning. Global Transformers like ViT compute attention globally, with the equivalent attention window size linearly proportional to the increased input image resolution. Local Transformers like Swin allow variable window sizes for flexibility and receptive field tuning. To handle the variable window sizes between pre-training and fine-tuning, bi-cubic interpolation was the previous common practice, but this approach is problematic for large resolution variations. In NLP, the relative position bias method proved beneficial compared to absolute position embedding. In vision, continuous convolution and variants utilize a meta network to handle irregular data points, which inspires the Log-CPB approach.

Swin Transformer is a general-purpose computer vision backbone that has achieved strong performance in various granular recognition tasks such as region-level object detection, pixel-level semantic segmentation, and image-level image classification. The architecture introduces several important visual priors into the vanilla Transformer encoder, including hierarchy, locality, and translation invariance. However, when scaling up from small to large sizes, the activation values at deeper layers increase dramatically — the discrepancy between layers with the highest and lowest amplitudes reaches an extreme value of 10^4. At an even larger size of 658 million parameters, training collapses entirely. Additionally, when directly testing pre-trained models at larger image resolutions through bi-cubic interpolation, accuracy drops from 81.7% to 68.7% when window size increases from 8x8 to 24x24.

The original Swin Transformer employs a pre-normalization configuration inherited from language Transformers, where layer normalization is applied at the beginning of each residual block. This creates a scaling issue: since the output of each residual block is directly added to the main branch without normalization, activation values accumulate layer by layer, and the amplitudes at deeper layers are significantly larger than those at early layers. To address this, residual post-normalization is proposed, which moves layer normalization from the beginning of each block to the end. In this configuration, the output of each residual block is normalized before merging back into the main branch, and the amplitude of the main branch does not accumulate as the network goes deeper. The activation amplitudes by this approach are much milder than in the original pre-normalization configuration.

For the largest models, an additional layer normalization layer on the main branch every 6 Transformer blocks is introduced to further stabilize training. The effectiveness is demonstrated empirically: while the activation values at deeper layers for the original Swin Transformer are almost exploded at large (L) size, those of the new version have much milder behavior. More critically, on a huge-size model (658M parameters), the self-supervised pre-training diverges using the original Swin Transformer, while it trains well with Swin Transformer V2. Table 6 shows systematic improvements across model scales: Tiny +0.2%, Small +0.4%, Base +0.5%, with greater benefits for larger models. A comparison with other normalization strategies (sandwich normalization) further validates the superiority of the proposed approach.

In the original self-attention mechanism, the similarity between pixel pairs is computed as a dot product of the query and key vectors. However, in large visual models, the authors observe that the learnt attention maps of some blocks and heads are frequently dominated by a few pixel pairs, leading to attention collapse. To address this, the scaled cosine attention replaces dot-product similarity with cosine similarity divided by a learnable scalar: Sim(q_i, k_j) = cos(q_i, k_j) / tau + B_ij, where tau is a learnable scalar (per attention head per layer, set no smaller than 0.01) and B_ij is the relative position bias. The cosine function is naturally normalized, so attention values are less likely to fall into extremes and the computation becomes irrelevant to the amplitudes of block inputs.

The combination of residual post-normalization and scaled cosine attention yields compounding benefits. Table 6 demonstrates that applying both techniques together achieves overall improvements of +0.2% (Tiny), +0.4% (Small), and +0.5% (Base) on ImageNet-1K, with the improvements being more beneficial for larger models. This trend suggests that the techniques will be even more critical at the Huge (658M) and Giant (3B) scales, where training without these modifications is infeasible due to instability or complete divergence. Compared to alternative normalization methods such as sandwich normalization, the proposed approach achieves superior accuracy (84.1% vs. 83.6% at Base size).

Rather than directly optimizing parameterized biases as in the original Swin Transformer, the continuous relative position bias (CPB) method employs a small meta-network on the relative coordinates to generate bias values: B(delta_x, delta_y) = G(delta_x, delta_y), where G is a two-layer MLP with a ReLU activation. Since the meta-network takes any continuous coordinates as input, a pre-trained model can freely transfer across window sizes by sharing weights of the meta-network. During inference, the bias values at each relative position can be pre-computed and stored as model parameters, such that the inference is identical to the original parameterized bias approach with zero additional computational cost.

When transferring across vastly different window sizes, large coordinate ranges require significant extrapolation beyond the training distribution. The key innovation is log-spaced coordinates: the relative coordinates are transformed as delta_x_hat = sign(delta_x) * log(1 + |delta_x|), and similarly for delta_y. This transformation dramatically reduces extrapolation requirements. For example, transferring from 8x8 to 16x16 windows, linear-spaced coordinates require extrapolation to 1.14x the original range, while log-spaced coordinates achieve only 0.33x — approximately 4 times smaller extrapolation ratio. Table 1 demonstrates the effectiveness: log-spaced CPB consistently outperforms parameterized position bias, particularly at larger window sizes. At W24, accuracy improves from 68.7% (parameterized) to 79.1% (log-spaced CPB) — a 10.4 percentage point gain. Testing with unseen window sizes (e.g., W12) can even yield +0.4% higher accuracy than the original pre-training window size.

Larger models are more data hungry. Previous large vision models relied on enormous labeled datasets such as JFT-3B (3 billion labeled images). Instead, the authors adopt the self-supervised pre-training method SimMIM, which uses a masked image modeling approach analogous to masked language modeling in NLP. Using this approach, they successfully trained a 3 billion-parameter Swin Transformer V2 model that achieves state-of-the-art results on 4 representative visual benchmarks, using only 70 million labelled images — 1/40 of JFT-3B. This demonstrates that effective self-supervised pre-training can substantially reduce the dependence on labeled data for training large vision models, mirroring the catalytic role that masked language modeling played for NLP scaling.

Training at such large scales also requires addressing prohibitively high GPU memory consumption. Three optimization techniques are employed. First, Zero-Redundancy Optimizer (ZeRO): model parameters and optimization states are split and distributed across multiple GPUs using the DeepSpeed framework (ZeRO stage-1), with little effect on training speed. Second, activation check-pointing: feature maps are recomputed during the backward pass instead of stored, significantly reducing memory while making training up to 30% slower. Third, sequential self-attention computation: for extreme scales (e.g., 1,536x1,536 images with 32x32 windows), self-attention is computed sequentially rather than in batch, applied to the first two stages with little impact on overall training speed. Together, these techniques enable training the 3B model on standard A100-40G GPUs.

On ImageNet-1K image classification, SwinV2-G achieves 90.17% top-1 accuracy on V1 and 84.0% on V2. The ImageNet-V2 result of 84.0% is +0.7% higher than the previous best (83.3% by ViT-G/14). For smaller model variants, SwinV2-B and SwinV2-L show +0.8% and +0.4% gains over their Swin V1 counterparts on ImageNet-1K. The SwinV2-G model is the largest dense vision model to date, with 3 billion parameters, trained on the privately collected ImageNet-22K-ext dataset with 70 million images. Pre-training uses 192x192 resolution, with self-supervised SimMIM pre-training for 20 epochs followed by supervised pre-training for 30 epochs.

On COCO object detection, SwinV2-G achieves 63.1 box AP and 54.4 mask AP on test-dev, which is +1.8/+1.4 higher than the previous best results (61.3/53.0 by SoftTeacher). This is a particularly significant result because it demonstrates that scaling up vision models is beneficial for dense vision recognition tasks, not just image classification. The training pipeline includes an additional detection pre-training phase using the Objects365 v2 dataset between classification pre-training and COCO fine-tuning. Testing with different window sizes (32x32 vs. 48x48) provides additional gains, showcasing the benefit of the Log-CPB transfer capability.

On ADE20K semantic segmentation, SwinV2-G achieves 59.9 mIoU on the validation set, +1.5 higher than the previous best (58.4 by BEiT). Using larger window sizes at test time provides an additional +0.2 gain, likely benefiting from the effective Log-CPB transfer. On Kinetics-400 video action classification, SwinV2-G achieves 86.8% top-1 accuracy, +1.4% higher than the previous best (85.4% by TokenLearner). Similarly, larger window sizes at test time yield an additional +0.2% gain. These results across four fundamentally different vision tasks — classification, detection, segmentation, and video understanding — collectively demonstrate the generality of the approach. The consistent benefits of test-time window size adjustment further validate the practical utility of Log-CPB.

Ablation studies systematically validate each component. For residual post-normalization and scaled cosine attention, Table 6 shows that both techniques individually contribute to accuracy gains, with compounding benefits when combined. The improvements grow with model size (+0.2% Tiny, +0.4% Small, +0.5% Base), confirming the scale-dependent nature of the benefit. Table 7 compares normalization strategies: the proposed approach (84.1% at Base) outperforms pre-norm (83.6%), sandwich norm, and standard post-norm. For window resolution transfer, the most striking result is that log-spaced CPB maintains 78.9% accuracy at W24 without fine-tuning, versus 68.7% for parameterized bias — demonstrating robust generalization to unseen window sizes. Fine-tuning further closes the gap, with log-spaced CPB achieving the best results across all tested window configurations.

This paper presents techniques for scaling Swin Transformer up to 3 billion parameters and making it capable of training with images of up to 1,536x1,536 resolution. The key contributions include residual-post-norm and scaled cosine attention for stable capacity scaling, and log-spaced continuous position bias for effective resolution transfer. Combined with SimMIM self-supervised pre-training, the model achieves state-of-the-art performance on ImageNet-V2 image classification (84.0%), COCO object detection (63.1/54.4 box/mask AP), ADE20K semantic segmentation (59.9 mIoU), and Kinetics-400 video classification (86.8%) — while requiring 40x less labelled data than previous approaches.

By scaling up both capacity and resolution of vision models with strong performance on general vision tasks, just like a good language model's performance on general NLP tasks, the authors aim to stimulate more research in this direction so that we can eventually close the capacity gap between vision and language models and facilitate the joint modeling of the two domains. The demonstrated techniques — particularly the log-spaced continuous position bias and the synergy between self-supervised pre-training and architectural innovations — establish a foundation for future scaling efforts in visual representation learning.