This paper introduces Dense Prediction Transformers (DPT), an architecture that leverages vision transformers in place of convolutional networks as backbone for dense prediction tasks. The model reassembles tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combines them using a convolutional decoder. The key advantage is that vision transformers maintain a constant and relatively high resolution throughout the computational process and have a global receptive field at every stage. DPT achieves a 28% improvement in relative performance on monocular depth estimation and sets new state-of-the-art results, producing "more fine-grained and globally coherent predictions" compared to fully-convolutional architectures.

Dense prediction tasks such as monocular depth estimation and semantic segmentation require generating per-pixel outputs from input images. Dominant approaches use convolutional neural networks with encoder-decoder architectures, where the encoder progressively downsamples the input, losing fine-grained spatial information. While skip connections and feature pyramid networks partially mitigate this loss, the fundamental issue remains: "downsampling has distinct drawbacks that are particularly salient in dense prediction tasks: feature resolution and granularity are lost." The authors argue that vision transformers offer a natural solution because they maintain constant resolution throughout processing and have global receptive fields at every stage.

Encoder-decoder architectures for dense prediction, pioneered by FCN and refined by U-Net and DeepLab, have become the standard paradigm. MiDaS achieved strong zero-shot monocular depth estimation by training on multiple mixed datasets with a ResNeXt-101 encoder. Vision Transformer (ViT) demonstrated that a pure transformer encoder applied to sequences of image patches can match or exceed CNNs for classification, but its adaptation to dense prediction is non-trivial because the output tokens lack explicit spatial structure. SETR made an initial attempt at using ViT for semantic segmentation but did not fully exploit the representation capacity of intermediate transformer layers.

DPT uses a Vision Transformer (ViT) as the encoder backbone. Three variants are employed: ViT-Base (12 layers, patch size p=16), ViT-Large (24 layers, D=1024), and ViT-Hybrid (ResNet50 embedding + 12 transformer layers). Tokens from four evenly-spaced transformer layers are extracted — for ViT-Base with 12 layers, tokens are taken from layers {3, 6, 9, 12}. Unlike CNN encoders that produce feature maps at decreasing resolutions, all extracted token sets have the same spatial resolution (H/p x W/p), preserving fine-grained information that would be lost in a CNN's deeper layers.

The Reassemble operation converts 1D token sequences back into 2D spatial feature maps at various resolutions. It consists of three steps: (1) Read: handles the readout token through one of three strategies — ignore, add, or project (concatenate and project); (2) Concatenate: reshapes the 1D token sequence into a 2D feature map at H/p x W/p resolution; (3) Resample: applies 1x1 convolutions to adjust channel dimensions followed by 3x3 transposed convolutions (with stride 2) or strided convolutions to produce feature maps at four resolutions: H/4, H/8, H/16, and H/32. This creates a multi-scale feature representation analogous to CNN feature pyramids.

The reassembled multi-scale features are combined using a RefineNet-based fusion module. Starting from the lowest resolution (H/32), features are progressively upsampled and fused with higher-resolution features using residual convolution units and multi-resolution fusion blocks. The final output is at H/2 resolution for depth estimation and H/1 for semantic segmentation. This decoder is deliberately kept lightweight to emphasize that the representation quality of the transformer encoder, rather than the decoder complexity, drives the performance gains.

On monocular depth estimation, DPT-Large achieves 28% relative improvement in zero-shot transfer, including 13.2% on DIW and 31.2% on ETH3D compared to the CNN-based MiDaS baseline. After fine-tuning, DPT sets new state-of-the-art on NYUv2 (0.110 AbsRel) and KITTI (0.062 AbsRel). On semantic segmentation, DPT-Hybrid achieves 49.02% mIoU on ADE20K, exceeding the prior best of 48.36%, and 60.46% mIoU on Pascal Context, setting a new state-of-the-art. Qualitative analysis reveals that DPT produces "finer-grained delineations of object boundaries" and more globally consistent depth maps — objects at similar distances receive more uniform depth values. Inference latency is comparable: 35ms for DPT-Large vs. 32ms for MiDaS.

Dense Prediction Transformers demonstrate that vision transformers can effectively serve as backbones for dense prediction tasks, producing "more fine-grained and globally coherent predictions when compared to fully-convolutional architectures." The key architectural contributions — the reassemble operation and multi-scale fusion decoder — enable seamless integration of ViT into existing dense prediction frameworks. The consistent improvements across depth estimation and semantic segmentation validate that the constant resolution and global receptive field of transformers are particularly beneficial for tasks requiring per-pixel understanding. The results suggest that transformer-based backbones will become the dominant paradigm for dense prediction.