Recently, Vision Transformer and its variants have shown great potential in various computer vision tasks. The ability of self-attention to model both short- and long-range visual dependencies is the key. However, existing approaches either use coarse-grained global attention sacrificing fine-grained local details, or fine-grained local attention at the cost of long-range modeling. In this work, the authors propose focal self-attention, a new mechanism where each token attends its closest surrounding tokens at fine granularity and tokens far away at coarse granularity. This enables efficient capture of both short- and long-range visual dependencies. With focal self-attention, the proposed Focal Transformer achieves superior performance on image classification (83.8% top-1 on ImageNet), object detection, and semantic segmentation.

Vision Transformers have emerged as powerful alternatives to convolutional networks. The core strength lies in self-attention's ability to model both short- and long-range dependencies. Yet this capability carries substantial computational costs when processing high-resolution feature maps. Existing approaches diverge into two strategies: coarse-grained global attention (such as downsampled tokens in PVT) sacrificing local detail, or fine-grained local attention (such as fixed windows in Swin Transformer) limiting long-range modeling. The authors observe that full self-attention ViT models indeed learn to attend local surroundings and global contexts simultaneously, suggesting both interaction types are necessary and should be captured in a single attention mechanism.

To this end, the authors propose focal self-attention where each query token attends its closest surrounding tokens at fine granularity and tokens far away at coarse granularity. This is achieved through multi-level sub-window pooling: at each focal level, the feature map is partitioned into sub-windows of increasing sizes and pooled to create multi-granularity representations. The query then attends keys and values from all levels simultaneously, capturing both fine-grained local patterns and coarse-grained global context in a single attention operation. This design maintains receptive field coverage comparable to global attention while achieving computational complexity linear in spatial dimensions.

Prior approaches to reducing attention complexity fall into two categories. The first employs coarse-grained global self-attention by attending downsampled or summarized tokens, as seen in PVT's spatial-reduction attention and Twins' global sub-sampled attention. The second uses fine-grained local attention within constant window sizes, exemplified by Swin Transformer's shifted window mechanism. Both strategies involve fundamental trade-offs: global methods lose fine-grained spatial resolution, while local methods require additional mechanisms (window shifting, relative position bias) to enable cross-window communication. The focal self-attention represents the first reconciliation of both approaches in a single transformer layer.

The design is motivated by a key empirical observation: visualizing attention patterns in full self-attention ViT reveals that attention heads naturally develop both local and global patterns. Some heads focus narrowly on immediate neighbors, while others attend broadly across the image. This suggests that an ideal attention mechanism should explicitly model both scales. Furthermore, ablation studies show that removing either local-only attention (82.2% to 80.1%) or global-only attention (82.2% to 81.5%) degrades performance, confirming the complementary nature of local and global interactions.

The focal self-attention mechanism operates at the window level. For each focal level l, the input feature map is partitioned into sub-windows of size s_w^l, then pooled using linear layers to create multi-granularity representations. Query tokens attend keys and values extracted from multiple focal levels simultaneously. For a given query window, surrounding regions are sampled at progressively coarser granularities — the further away the region, the coarser the granularity. Three key parameters specify focal attention: focal levels (L) controlling the number of granularity levels, focal window size (s_w^l) specifying the sub-window dimension at level l, and focal region size (s_r^l) defining the extent of attended regions. The overall computational complexity is O((L + sum(s_r^l)^2)(MN)d), which is linear in spatial dimensions rather than quadratic.

The Focal Transformer follows a multi-scale design with four stages. Input images undergo 4x4 patch embedding, then pass through stages where spatial resolution decreases by factor 2 while feature dimension increases by 2 at transitions. Each stage contains focal transformer layers processing feature maps at decreasing resolutions. The model comes in three variants: Focal-Tiny (29M parameters), Focal-Small (51M), and Focal-Base (90M), designed to match the parameter counts of Swin Transformer variants for fair comparison.

On ImageNet-1K image classification, Focal-Tiny achieves 82.2% top-1 accuracy (+1.0% vs. Swin-Tiny) at similar parameters. Focal-Small reaches 83.5% and Focal-Base achieves 83.8%, consistently outperforming comparable models. For COCO object detection, Focal-Tiny improves over Swin-Tiny by +1.7 box mAP and +1.2 mask mAP with Mask R-CNN. Testing across six detection methods (Cascade R-CNN, ATSS, RepPoints, Sparse R-CNN) demonstrates consistent gains of 1.0-2.3 mAP. On ADE20K semantic segmentation, Focal-Large achieves 55.4 mIoU (+1.9 vs. Swin-Large), establishing state-of-the-art performance. Notably, removing window shifting from Focal Transformers shows minimal degradation (unlike Swin), confirming that focal attention inherently captures cross-window information without requiring additional mechanisms.

Focal self-attention successfully reconciles efficiency with modeling capacity by performing local self-attention at fine granularity and global self-attention at coarse granularity. The approach generalizes across classification, detection, and segmentation with consistent improvements over state-of-the-art baselines. The authors acknowledge that computational overhead remains a consideration for practical deployment, suggesting depth reduction and broader architectural exploration as future directions for making focal attention more efficient.