The authors present Multiscale Vision Transformers (MViT) for video and image recognition, connecting multiscale feature hierarchies with transformer models. The architecture features multiple channel-resolution scale stages that hierarchically expand channel capacity while reducing spatial resolution. This enables early layers to operate on high-resolution features with compact channel dimensions, while deeper layers process coarse, complex features with expanded channels. A key innovation is Multi Head Pooling Attention (MHPA), which applies pooling operators to query, key, and value tensors for flexible resolution modeling within transformer blocks. MViT achieves state-of-the-art results on Kinetics-400 video classification (81.2%) with 5-10x fewer FLOPs than concurrent ViT-based models, and competitive ImageNet performance (84.8%) with 1.7x fewer FLOPs than DeiT, all without large-scale pre-training.

Vision Transformers (ViT) have demonstrated impressive results on image classification, but their columnar architecture processes all layers at the same spatial resolution and channel dimension. This is fundamentally at odds with the multiscale nature of visual signals: natural images contain features at multiple scales, and biological visual systems process them hierarchically. In convolutional networks, the progressive reduction in spatial resolution alongside increase in channel capacity — a multiscale pyramid — has been a cornerstone of successful architectures from VGGNet to ResNet. The authors argue that this multiscale principle should be the foundation of transformer architectures for vision, not an afterthought.

For video understanding, the multiscale principle is even more critical. Video data is extremely dense — a short clip contains millions of spatiotemporal tokens. Processing all tokens at full resolution through standard self-attention is computationally prohibitive. MViT addresses this by starting with thin channel dimensions at high spatiotemporal resolution in early layers and gradually expanding channels while reducing resolution through pooling. This creates a multiscale pyramid of features where computational complexity remains roughly constant across stages, enabling efficient processing of dense visual data without sacrificing the ability to capture fine-grained patterns.

Prior vision transformers maintain constant resolution throughout all layers. ViT and DeiT process fixed-size tokens from patch embedding to output. For video, TimeSformer and ViViT extend ViT to spatiotemporal tokens but maintain the same columnar design, requiring massive pre-training on ImageNet-21K. Concurrent works like PVT and Swin Transformer introduce multiscale features for dense prediction, but they focus on image tasks and do not address the spatiotemporal nature of video. In the CNN domain, SlowFast networks demonstrated the value of multi-rate temporal processing for video. MViT uniquely combines multiscale spatial and temporal modeling within a pure transformer architecture, without relying on large-scale pre-training.

MViT implements a progressive channel-resolution trade-off across stages. The network begins with a small channel dimension (e.g., 96) at high spatiotemporal resolution and progressively doubles the channel dimension while halving the spatial resolution at stage transitions. This design is motivated by the observation that early visual processing requires high spatial resolution to capture fine-grained local patterns (edges, textures), while deeper processing requires rich channel capacity to encode complex semantic concepts. The normalized skip connections at stage transitions accommodate the changing dimensions, and separate space and time positional embeddings (rather than joint spatiotemporal) provide position information with factored complexity.

The core innovation is Multi Head Pooling Attention (MHPA), which enables flexible resolution modeling within a single transformer block. Unlike standard Multi-Head Attention that maintains constant sequence length, MHPA applies learnable pooling operators to query (Q), key (K), and value (V) tensors with configurable kernel sizes, strides, and padding. The pooling reduces the sequence length of K and V, decreasing the attention computation cost, while optionally also pooling Q to produce downsampled output for stage transitions. This mechanism serves dual purposes: it reduces computational cost within stages and enables resolution changes across stages, all within the attention mechanism itself rather than requiring separate downsampling layers.

The architecture follows a channel-resolution scaling principle: as spatial resolution decreases by factor s, channel dimension increases by factor s, keeping the total computation per stage roughly constant. The model is instantiated with MViT-B (36.6M parameters), starting from 96 channels at 16x temporal and spatial resolution, expanding to 768 channels at reduced resolution through four stages. For video inputs, the initial patch embedding produces spatiotemporal tokens by non-overlapping 3D patches. A wider variant (MViT-B-24-wide) achieves the best ImageNet accuracy. The entire design avoids large-scale pre-training requirements — MViT is trained from scratch on the target dataset, which is a significant practical advantage.

On Kinetics-400 video classification, MViT-B achieves 81.2% top-1 accuracy, surpassing concurrent transformer models that require ImageNet-21K pre-training, with 5-10x lower computation and parameters. A crucial temporal modeling experiment reveals: shuffling input frames causes a significant 7.1% accuracy decay for MViT, but only 0.1% for standard ViT. This confirms that MViT genuinely captures temporal dynamics while standard ViT performs mere "bag-of-frames" classification. On ImageNet image classification, MViT-B-24-wide reaches 84.8% top-1 accuracy with 1.7x fewer FLOPs than DeiT-B at higher resolution. Ablation studies confirm that MHPA, separate positional embeddings, and the multiscale design each contribute essential improvements.

MViT demonstrates that multiscale feature hierarchies are essential for effective vision transformers, particularly for dense, high-dimensional data like video. The Multi Head Pooling Attention mechanism enables flexible resolution modeling within the attention framework, achieving state-of-the-art video recognition with dramatically fewer resources and competitive image classification without pre-training. The channel-resolution scaling principle aligns transformer architectures with classical vision principles of multiscale processing, suggesting that the most effective vision transformers will be those that embrace, rather than ignore, the multiscale nature of visual data.