Vision Transformer (ViT) has shown promising performance on image classification but requires large-scale pre-training datasets (e.g., JFT-300M) to achieve competitive results. This paper identifies two key limitations: the simple tokenization of input images fails to model the important local structure such as edges and lines, leading to low training sample efficiency; and the redundant attention backbone of ViT is not efficient enough for vision tasks when training from scratch. The authors propose Tokens-to-Token ViT (T2T-ViT), which progressively tokenizes images through aggregating neighboring tokens and achieves 81.5% top-1 accuracy on ImageNet training from scratch.

The original ViT splits an image into non-overlapping 16x16 patches and linearly projects each patch into a token. This hard splitting destroys the local continuity and structural information of the image, such as edges, corners, and texture patterns that convolutional neural networks naturally capture through their hierarchical receptive fields. When trained solely on ImageNet-1K (1.3M images), ViT-Base achieves only 77.9% top-1 accuracy, significantly below DeiT-Base's 81.8% which uses extensive data augmentation.

Furthermore, the authors analyze the attention maps of ViT and observe that many attention heads in the later layers exhibit high similarity, indicating redundancy in the backbone design. This redundancy means the model wastes parameters on repeated computations rather than learning diverse, complementary representations. The authors argue that a more compact and efficient backbone architecture, inspired by CNN design principles, can reduce this redundancy while maintaining representational power.

Vision Transformers have rapidly evolved since the original ViT demonstrated that pure transformer architectures can achieve competitive image classification performance. DeiT introduced knowledge distillation and advanced data augmentation strategies to improve ViT's training efficiency on ImageNet. Meanwhile, CNN architectures like ResNet and EfficientNet have long benefited from carefully designed hierarchical structures that progressively build local-to-global representations — a principle largely absent in vanilla ViT.

The T2T module performs progressive tokenization through iterative token restructuring. In each T2T step, the module first applies a transformer layer to model relationships among all tokens, then reshapes the output tokens back into a spatial image format and unfolds overlapping patches from neighboring tokens to aggregate local information. This process progressively reduces the token length while enriching each token with surrounding structural information, effectively encoding local patterns like edges and textures that simple patch splitting misses.

Specifically, the T2T module uses a soft split operation with overlapping windows. Given tokens reshaped into a 2D spatial map, a sliding window with kernel size k and stride s (where s < k) generates overlapping patches that are concatenated along the channel dimension. After two T2T iterations, the token count is reduced from the original patch count while each token now encodes rich local context. The final tokens are then fed into the efficient transformer backbone.

For the transformer backbone, the authors explore architectures inspired by CNN design principles including dense connections (DenseNet-style), channel attention (SE-Net-style), and ghost operations (GhostNet-style). Through systematic comparison, a deep-narrow structure with fewer channels but more layers proves most effective. This design reduces the attention head redundancy observed in vanilla ViT while maintaining comparable representational capacity with significantly fewer parameters.

On ImageNet-1K, T2T-ViT-14 achieves 81.5% top-1 accuracy with only 21.5M parameters, comparable to ResNet-50 (25.6M parameters, 76.1% accuracy) and DeiT-Small (22.1M parameters, 79.8% accuracy). T2T-ViT-24 further reaches 82.3% accuracy with 64.1M parameters. Ablation studies show that the T2T module alone improves accuracy by +2.1% over simple patch embedding, and the efficient backbone design contributes an additional +1.4% improvement. On transfer learning tasks, T2T-ViT maintains competitive performance on CIFAR-10, CIFAR-100, and Oxford Flowers.

T2T-ViT addresses two fundamental limitations of vanilla ViT through progressive tokenization that models local structure and an efficient deep-narrow backbone that reduces attention redundancy. The result is a vision transformer that achieves CNN-level performance when training from scratch on ImageNet, without requiring massive pre-training datasets or knowledge distillation. This work demonstrates that bridging CNN design wisdom with transformer architectures yields models that are both data-efficient and computationally practical.