Most recent semantic segmentation methods adopt a fully convolutional network (FCN) with an encoder-decoder architecture. The encoder progressively reduces spatial resolution and learns more abstract/semantic visual concepts with larger receptive fields. Since context modeling is critical for segmentation, the latest efforts have been focusing on increasing the receptive field. This paper proposes SETR, which treats semantic segmentation as a sequence-to-sequence prediction task. Specifically, an image is treated as a sequence of patches, and a pure transformer (without convolution and resolution reduction) is deployed to encode the image as a sequence of feature representations with global context modeled in every layer of the transformer.

Fully Convolutional Networks (FCNs) have been the dominant paradigm for semantic segmentation since their introduction. These approaches typically employ an encoder-decoder structure where the encoder, often a deep convolutional neural network pre-trained on ImageNet, progressively reduces spatial resolution while increasing feature abstraction. The decoder then recovers spatial details through upsampling. A fundamental limitation is that the receptive field grows only linearly with network depth, making it difficult to capture long-range dependencies efficiently.

The Transformer architecture, originally designed for natural language processing, employs self-attention mechanisms that model dependencies between all positions in a sequence regardless of their distance. The recent success of Vision Transformer (ViT) on image classification demonstrates that a pure transformer applied to sequences of image patches can perform very well on recognition tasks. This raises a natural question: can we rethink the semantic segmentation task from a sequence-to-sequence perspective, replacing convolution-based encoders entirely?

Prior efforts to enlarge receptive fields include dilated/atrous convolutions, Atrous Spatial Pyramid Pooling (ASPP), and non-local operations. While these methods partially address the limited receptive field problem, they are still built upon the FCN backbone and inherit its fundamental constraints. Concurrently, DETR demonstrated that transformers can tackle object detection as a set prediction problem, but it still relies on a CNN backbone for feature extraction. The recent ViT showed that a pure transformer without any convolutional layers achieves competitive classification performance, yet its application to dense prediction tasks like segmentation remains unexplored.

Given an image of size H x W x 3, SETR first splits it into a sequence of flattened 2D patches of size p x p, producing a sequence of HW/p^2 tokens. Each patch is linearly projected into an embedding of dimension d, and learnable 1D position embeddings are added. The resulting sequence is then fed into a standard Transformer encoder consisting of L layers of Multi-Head Self-Attention (MHSA) and Feed-Forward Networks (FFN). Critically, every layer performs global self-attention over the entire patch sequence, meaning that each patch attends to all other patches regardless of spatial distance from the very first layer.

The authors propose two simple decoder designs: Progressive UPsampling (PUP) and Multi-Level feature Aggregation (MLA). PUP progressively upsamples the Transformer output by a factor of 2 in each step using bilinear interpolation followed by convolution, restoring the original resolution in a multi-stage fashion. MLA aggregates intermediate features from multiple Transformer layers (e.g., layers 6, 12, 18, 24) and fuses them through channel attention. A third variant, SETR-Naive, simply reshapes and upsamples the final layer output in one step. The key insight is that even these simple decoders, combined with the powerful Transformer encoder, achieve state-of-the-art results.

Experiments are conducted on three major benchmarks: ADE20K (150 categories), Pascal Context (59 categories), and Cityscapes (19 categories). The Transformer encoder uses ViT-Large with 24 layers, hidden dimension 1024, and 16 attention heads, pre-trained on ImageNet-21K. On ADE20K, SETR-MLA achieves 50.28% mIoU, surpassing all previous methods including HRNet+OCR (45.66%) and DeepLabV3+ (45.47%). On Pascal Context, SETR-MLA reaches 55.83% mIoU, again establishing a new state of the art. On Cityscapes test, SETR-PUP achieves 82.2% mIoU, competitive with the best methods.

Ablation studies reveal several key findings: increasing the number of Transformer layers from 12 to 24 improves mIoU by approximately 2%. MLA consistently outperforms PUP and Naive decoders, confirming the value of multi-level feature aggregation. Position embedding analysis shows that learned 1D position embeddings perform comparably to 2D variants, suggesting the Transformer implicitly learns spatial structure. Importantly, features from intermediate layers (e.g., layer 15) sometimes outperform the final layer, indicating that not all useful information propagates to the deepest layers.

This paper presents SETR, an alternative perspective that reformulates semantic segmentation as a sequence-to-sequence prediction task using a pure Transformer encoder. By treating images as sequences of patches and applying global self-attention in every layer, SETR overcomes the fundamental receptive field limitation of FCN-based methods. Combined with simple decoder designs, SETR achieves new state-of-the-art results on ADE20K (50.28% mIoU) and Pascal Context (55.83% mIoU). The results demonstrate that the Transformer architecture offers a viable and powerful alternative to convolutional encoders for dense prediction tasks.