This paper presents a new vision Transformer, called Swin Transformer, that capably serves as a general-purpose backbone for computer vision. Challenges in adapting Transformer from language to vision arise from differences between the two domains, such as large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. To address these differences, the authors propose a hierarchical Transformer whose representation is computed with shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection. This hierarchical architecture has the flexibility to model at various scales and has linear computational complexity with respect to image size. Swin Transformer achieves 87.3% top-1 accuracy on ImageNet-1K, 58.7 box AP and 51.1 mask AP on COCO test-dev, and 53.5 mIoU on ADE20K val, surpassing previous state-of-the-art by significant margins.

Convolutional neural networks (CNNs) have served as the dominant backbone architectures in computer vision for the past decade. Beginning with AlexNet and its revolutionary performance on ImageNet, CNN architectures have evolved to become increasingly powerful through deeper layers, more extensive connections, and more sophisticated convolution forms. Meanwhile, the Transformer architecture, originally designed for natural language processing, has demonstrated remarkable performance for sequence-to-sequence modeling. Its adoption in vision—Vision Transformer (ViT)—has shown impressive results on image classification, yet the architecture faces fundamental challenges when applied to dense prediction tasks that require multi-scale feature maps.

The dominant challenge lies in the quadratic computational complexity of global self-attention with respect to image size, which makes it infeasible for high-resolution dense prediction tasks such as object detection and semantic segmentation. Furthermore, ViT produces single-resolution feature maps, lacking the multi-scale feature hierarchy that has proven essential for Feature Pyramid Networks (FPN) and similar dense prediction architectures. The authors argue that a general-purpose vision backbone must produce hierarchical feature representations with manageable computational cost, motivating the design of Swin Transformer.

CNN backbones such as VGG, ResNet, and EfficientNet have demonstrated that hierarchical feature extraction with progressively increasing receptive fields is highly effective for a wide range of visual recognition tasks. These architectures inherently produce multi-scale feature maps through successive pooling or strided convolution. The Vision Transformer (ViT) applied a standard Transformer encoder to non-overlapping image patches, achieving state-of-the-art image classification when pre-trained on very large datasets. However, ViT's isotropic architecture — maintaining the same resolution and feature dimension throughout — is incompatible with multi-scale frameworks like FPN. DeiT improved ViT's data efficiency through knowledge distillation, but did not address the architectural limitations for dense tasks.

The Swin Transformer builds hierarchical feature maps through a four-stage architecture. In the first stage, input images are split into non-overlapping 4x4 patches, each treated as a "token" with a raw feature dimension of 4x4x3 = 48. A linear embedding layer projects these to dimension C. Subsequent stages employ patch merging layers that concatenate features of each group of 2x2 neighboring patches and apply a linear layer to reduce the dimension to 2C, effectively downsampling the feature map resolution by 2x while doubling the channel dimension. This produces feature maps at resolutions of H/4, H/8, H/16, and H/32, mirroring the multi-scale structure of typical CNN backbones like ResNet.

The core innovation of Swin Transformer is the shifted window partitioning scheme. In standard window-based multi-head self-attention (W-MSA), the feature map is evenly partitioned into non-overlapping MxM windows (default M=7), and self-attention is computed independently within each window. This reduces computational complexity from quadratic O(n^2) for global attention to linear O(n) when M is fixed. Specifically, for a feature map of hw patches, global MSA costs 4hwC^2 + 2(hw)^2C, while window MSA costs only 4hwC^2 + 2M^2hwC. However, windowed attention lacks connections across windows, limiting the model's capacity.

To introduce cross-window connections without increasing computational cost, consecutive Swin Transformer blocks alternate between regular and shifted window partitioning. The shifted configuration displaces the windows by (floor(M/2), floor(M/2)) pixels from the regularly partitioned configuration. This design "provides connections among non-overlapping windows in the preceding layer, significantly enhancing modeling power." An efficient batch computation approach using cyclic shifting ensures that the number of batched windows remains the same, and a masking mechanism restricts self-attention within each sub-window to maintain correctness. Ablation studies confirm that shifted windows improve ImageNet top-1 accuracy by 1.1%, COCO box AP by 2.8, and ADE20K mIoU by 2.8 compared to non-shifted baselines.

The paper defines four model variants: Swin-T (tiny, C=96, layers={2,2,6,2}), Swin-S (small, C=96, layers={2,2,18,2}), Swin-B (base, C=128, layers={2,2,18,2}), and Swin-L (large, C=192, layers={2,2,18,2}). The complexity of Swin-T is designed to be similar to ResNet-50, and Swin-S/B are similar to ResNet-101. Each variant uses a window size of M=7, the query dimension of each head is d=32, and an expansion ratio of 4 is used in the MLP layers. Relative position bias is used instead of absolute position embeddings, which the authors show consistently outperforms absolute position embeddings across all three tasks.

On ImageNet-1K image classification, Swin-T achieves 81.3% top-1 accuracy, surpassing DeiT-S (79.8%) and ResNet-50 at comparable model sizes. With ImageNet-22K pre-training, Swin-L reaches 87.3% top-1 accuracy at 384x384 resolution. On COCO object detection using the HTC++ framework, Swin-L achieves 58.7 box AP and 51.1 mask AP, representing +2.7 box AP and +2.6 mask AP improvements over previous state-of-the-art. On ADE20K semantic segmentation, Swin-L reaches 53.5 mIoU, a +3.2 mIoU improvement. Notably, using Cascade Mask R-CNN, Swin-T achieves +3.4 to 4.2 box AP gains over ResNet-50, and on ADE20K, Swin-T obtains +5.3 mIoU higher than DeiT-S.

Ablation studies further validate key design choices. Comparing shifted vs. non-shifted windows on Swin-T: ImageNet accuracy improves from 80.2% to 81.3% (+1.1%), COCO box AP improves from 47.7 to 50.5 (+2.8 AP), and ADE20K mIoU improves from 43.3 to 46.1 (+2.8 mIoU). The relative position bias consistently outperforms absolute position embeddings across all tasks, indicating that the model benefits from "inductive bias that encourages certain translation invariance." The shifted window approach also runs 4.1x faster than sliding window variants while achieving comparable accuracy.

This paper presents Swin Transformer, a new vision Transformer that produces a hierarchical feature representation and has linear computational complexity with respect to input image size. Swin Transformer achieves state-of-the-art performance on COCO object detection and ADE20K semantic segmentation, significantly surpassing previous best methods. The key design element — shifted window-based self-attention — proves effective and efficient for visual modeling. The authors believe the unified architecture across vision and language could benefit both fields, potentially enabling joint modeling across domains. The strong performance across image classification, detection, and segmentation establishes Swin Transformer as a general-purpose backbone for computer vision.