In this work, we introduce Dual Attention Vision Transformers (DaViT), a simple yet effective vision transformer architecture that is able to capture global context while maintaining computational efficiency. The key idea is to alternate between spatial window attention and channel group attention in each transformer block. Spatial window attention captures local fine-grained features within non-overlapping windows, while channel group attention models global interactions by attending across spatial locations in a channel-grouped manner. Extensive experiments demonstrate that DaViT achieves state-of-the-art performance on ImageNet-1K classification, COCO object detection, and ADE20K semantic segmentation, outperforming existing vision transformers with comparable computational budgets.

Vision Transformers (ViTs) have achieved remarkable success in computer vision, demonstrating strong performance on image classification, object detection, and semantic segmentation. However, the quadratic computational complexity of global self-attention with respect to the number of tokens poses a significant challenge for processing high-resolution images. Various strategies have been proposed to address this limitation, including local window attention as used in Swin Transformer, dilated attention, and linear attention approximations. While effective at reducing computation, these approaches often sacrifice the ability to model long-range dependencies at every layer.

We observe that attention can be computed along two orthogonal dimensions: spatial and channel. Spatial attention computes relationships between tokens at different spatial positions within a local region, focusing on "where" to attend. Channel attention computes relationships across the feature channels at each spatial position, focusing on "what" features are important. By alternating between these two types of attention, we can effectively capture both local spatial patterns and global semantic information without incurring the quadratic cost of full global attention. This dual perspective provides a more comprehensive representation than either attention type alone.

Hierarchical vision transformers have become the dominant architecture paradigm. Swin Transformer introduced shifted window attention to enable cross-window information exchange. PVT and Twins use spatial reduction to decrease the key-value length in attention computation. CSWin Transformer proposes cross-shaped window attention for more efficient global modeling. Despite their effectiveness, most of these methods focus solely on spatial attention and do not explicitly model channel-wise interactions. In the CNN literature, channel attention mechanisms such as SE-Net and CBAM have demonstrated the importance of adaptive channel weighting. DaViT bridges these two lines of research by incorporating both spatial and channel attention within the transformer framework.

Beyond attention mechanism design, multi-scale feature extraction is another critical component of modern vision architectures. Methods like FPN and BiFPN have demonstrated the importance of combining features at different resolutions for dense prediction tasks. DaViT adopts a four-stage hierarchical structure with progressively reduced spatial resolution and increased channel dimensions, similar to the design philosophy of Swin and PVT. However, unlike these methods, DaViT's dual attention mechanism ensures that global context is captured at every stage through channel group attention, rather than relying on indirect cross-window mechanisms.

The DaViT architecture follows a hierarchical design with four stages. Each stage consists of multiple Dual Attention Transformer (DAT) blocks, where each block contains two sequential attention layers: a spatial window attention layer followed by a channel group attention layer. In the spatial window attention layer, the input feature map is partitioned into non-overlapping windows of fixed size (e.g., 7x7), and standard multi-head self-attention is applied within each window. This captures local spatial relationships with linear complexity relative to the input resolution. In the channel group attention layer, the feature channels are divided into groups, and attention is computed across all spatial positions within each channel group, capturing global spatial relationships with complexity proportional to the number of channel groups.

The channel group attention operates by reshaping the feature tensor from (B, H, W, C) to (B, G, N, C/G), where G is the number of groups and N = H x W is the number of spatial tokens. Attention is then computed along the spatial dimension N for each group independently, resulting in G independent attention maps that collectively cover all channels. This formulation allows each group to specialize in different semantic aspects while maintaining computational cost of O(G * N^2 * C/G) = O(N^2 * C), which is equivalent to standard global attention but with significantly smaller attention matrices. Between stages, patch merging layers reduce the spatial resolution by 2x and double the channel dimension, creating the hierarchical multi-scale structure necessary for dense prediction tasks.

We evaluate DaViT on three representative benchmarks. On ImageNet-1K classification, DaViT-Tiny (28.3M parameters) achieves 82.8% top-1 accuracy, outperforming Swin-T (81.3%), PVTv2-B2 (82.0%), and CSWin-T (82.7%). DaViT-Small (49.7M) reaches 84.2% accuracy, surpassing Swin-S (83.0%) and CSWin-S (83.6%). DaViT-Base (87.9M) achieves 84.6%, competitive with the best models at this scale. These results demonstrate that the dual attention mechanism consistently provides improvements across different model sizes.

On COCO object detection using Mask R-CNN, DaViT-Tiny achieves 45.0 box AP and 40.7 mask AP, outperforming Swin-T (43.7/39.8) by significant margins. On ADE20K semantic segmentation with UperNet, DaViT-Small achieves 49.4 mIoU, surpassing Swin-S (48.5) and CSWin-S (49.2). Ablation studies confirm the complementary nature of the two attention types: removing channel group attention decreases ImageNet accuracy by 0.9%, while removing spatial window attention causes a 1.3% drop, demonstrating that both attention mechanisms are essential and contribute unique information.

We have introduced DaViT, a vision transformer architecture that combines spatial window attention with channel group attention to achieve both local fine-grained feature extraction and global context modeling. Through extensive experiments on classification, detection, and segmentation benchmarks, we demonstrate that DaViT consistently outperforms existing vision transformers with comparable computational budgets. The dual attention design provides a simple and effective approach to bridging the gap between computational efficiency and representational capacity in vision transformers. We hope that DaViT inspires further exploration of orthogonal attention designs that jointly optimize for different aspects of visual understanding.