Although Transformer has recently demonstrated encouraging results in computer vision, it still has limitations when used as a backbone for dense prediction tasks such as object detection and segmentation. Unlike Vision Transformer (ViT), which produces single-scale, low-resolution outputs, the authors introduce Pyramid Vision Transformer (PVT), which overcomes these difficulties by generating multi-scale feature maps through a progressive shrinking pyramid while maintaining computational efficiency via Spatial-Reduction Attention (SRA). PVT can be plugged into many representative dense prediction pipelines, achieving significant improvements over ResNet on COCO detection and ADE20K segmentation.

Convolutional Neural Networks (CNNs) have dominated computer vision for years, with architectures like ResNet and ResNeXt serving as standard backbones for both image classification and dense prediction tasks. Recently, Vision Transformer (ViT) demonstrated that a pure transformer architecture can achieve competitive classification performance. However, exploring convolution-free transformer backbones for dense prediction remains rarely studied. ViT produces only single-scale, low-resolution feature maps, making it unsuitable as a direct replacement for CNN backbones in detection and segmentation frameworks that require multi-scale feature pyramids.

To address this gap, the authors introduce Pyramid Vision Transformer (PVT), a pure transformer backbone that generates multi-scale feature maps for dense prediction. PVT inherits the advantages of both CNN and Transformer: like CNNs, it produces hierarchical feature maps at four stages with progressively decreasing resolutions (stride 4, 8, 16, 32); like Transformers, it captures long-range dependencies through self-attention. The key innovation is the Spatial-Reduction Attention (SRA) mechanism, which reduces the spatial dimensions of keys and values before computing attention, achieving significant computational savings without sacrificing performance.

CNN backbones for dense prediction have evolved from VGGNet to ResNet, ResNeXt, and EfficientNet, all sharing a common pyramidal design that produces multi-scale features. For dense prediction, frameworks such as Feature Pyramid Network (FPN), RetinaNet, and Mask R-CNN rely heavily on these multi-scale features. In the Transformer domain, ViT and DeiT have demonstrated strong classification performance but maintain a columnar, single-resolution architecture throughout all layers, making them incompatible with multi-scale detection and segmentation pipelines.

PVT consists of four stages, each producing feature maps at different scales. In Stage 1, the input image is divided into 4x4 patches, which are linearly embedded into tokens. These tokens are processed by a Transformer encoder, and the output is reshaped to form a feature map at 1/4 resolution. Subsequent stages further reduce the spatial resolution by a factor of 2 through patch embedding layers, yielding feature maps at 1/8, 1/16, and 1/32 of the input resolution. This progressive shrinking pyramid mirrors the multi-scale design of CNN backbones while operating entirely through attention mechanisms.

Standard Multi-Head Attention (MHA) has computational complexity of O(n^2) where n is the number of tokens, making it prohibitively expensive for high-resolution feature maps. To address this, the authors propose Spatial-Reduction Attention (SRA), which applies a spatial reduction operation to keys and values before computing attention. Specifically, the key and value sequences are reshaped into 2D feature maps and downsampled by a factor R_i using a linear layer, reducing the sequence length from n to n/R_i^2. This achieves R_i^2 times lower computational and memory costs compared to standard MHA, making it feasible to process high-resolution feature maps in early stages.

Unlike CNNs that use convolutional strides to reduce spatial resolution, PVT employs patch embedding layers at the beginning of each stage to progressively shrink the feature map. At stage i, the feature map from the previous stage is divided into patches of size P_i x P_i and linearly projected to form new token sequences. This enables flexible control over the resolution and channel dimensions at each stage, allowing the model to allocate more computational resources to early, high-resolution stages where fine-grained spatial information is critical. The reduction ratios R_i are set inversely to the stage depth — larger reductions in early stages (R_1=8) and smaller in later stages (R_4=1) — balancing computation across stages.

PVT is evaluated across three major benchmarks. On ImageNet classification, PVT-Large achieves 81.7% top-1 accuracy, competitive with ResNeXt-101. For COCO object detection, replacing ResNet-50 with PVT-Small in RetinaNet yields +4.1 AP improvement (36.7 to 40.4); in Mask R-CNN, PVT-Small achieves +3.9 AP_box and +3.4 AP_mask improvements over ResNet-50. On ADE20K semantic segmentation with Semantic FPN, PVT-Large achieves 42.1 mIoU, surpassing ResNet-101 (38.8 mIoU) by 3.3 points. These results demonstrate that PVT can serve as a versatile, convolution-free backbone competitive with well-established CNN architectures across diverse dense prediction tasks.

This paper presents PVT, demonstrating that a pure transformer architecture with pyramid structure can effectively replace CNN backbones for dense prediction tasks. Through progressive shrinking and Spatial-Reduction Attention, PVT generates multi-scale features efficiently while capturing long-range dependencies. Extensive experiments on classification, detection, and segmentation validate its versatility. The authors note that transformer-based models remain in early development compared to mature CNN designs, and anticipate further improvements as the community develops better training strategies, data augmentation techniques, and architectural innovations for vision transformers.