We present VideoMamba, addressing the dual challenges of local redundancy and global dependencies in video understanding by innovatively adapting the Mamba architecture to the video domain. Unlike 3D convolutions that have limited receptive fields and video Transformers that suffer from quadratic complexity, VideoMamba's linear-complexity operator enables efficient long-term modeling crucial for high-resolution long video understanding. Extensive evaluations reveal VideoMamba's four core abilities: (1) Scalability without extensive pretraining via a novel self-distillation technique; (2) Sensitivity for short-term action recognition with fine-grained motion differences; (3) Superiority in long-term video understanding; and (4) Compatibility with other modalities in multi-modal contexts.

The adaptation of State Space Models (SSMs) to video understanding represents a paradigm shift from attention-based temporal modeling. While attention mechanisms provide global context, their O(n^2) scaling with sequence length makes them impractical for long videos where the number of tokens easily reaches hundreds of thousands. SSMs offer O(n) complexity through their recurrent formulation while maintaining the ability to capture long-range dependencies through learned state transitions. VideoMamba is the first comprehensive exploration of this paradigm for video understanding, providing both architectural innovations and practical guidelines.

The explosive growth of video content — with over 500 hours uploaded to YouTube every minute — has created an urgent need for scalable video understanding systems. Applications ranging from content moderation and recommendation to autonomous driving and robotics require models that can efficiently process video at scale. However, the computational cost of existing methods has limited deployment to short clips (typically 4-16 seconds), leaving the vast majority of long-form video content underserved by current AI systems.

Video understanding presents unique challenges compared to image understanding: videos contain massive spatio-temporal redundancy (adjacent frames are highly similar) while also requiring modeling of long-range temporal dependencies (understanding a story or complex action sequence). 3D CNNs handle local patterns well but cannot capture long-range dependencies efficiently. Video Transformers can model global interactions but their O(n^2) complexity makes them prohibitively expensive for long videos. The recent State Space Model (SSM) paradigm, exemplified by Mamba, offers a compelling alternative with linear complexity while maintaining the ability to model long-range dependencies through its selective scan mechanism.

The emergence of VideoMamba is situated within a broader trend of efficient sequence modeling gaining traction across machine learning. The success of Mamba in natural language processing — where it achieves competitive results with Transformers at a fraction of the computational cost for long sequences — naturally raised the question of whether similar benefits transfer to visual domains. The answer, as demonstrated by Vim for images and now VideoMamba for videos, is affirmative but nuanced: SSMs excel particularly in scenarios where sequence length is the computational bottleneck, which is precisely the case for video understanding where temporal extent creates orders-of-magnitude more tokens than static images.

The evolution of video understanding architectures spans three generations. 3D CNNs (C3D, I3D, SlowFast) extended 2D convolutions to the temporal dimension but were limited to local temporal windows. Video Transformers (ViViT, TimeSformer, Video Swin) introduced global attention but required various efficiency tricks such as factorized attention, windowed attention, or sparse sampling to manage complexity. Mamba and its visual variant Vim (Vision Mamba) recently demonstrated that SSMs can achieve competitive results on image tasks with linear complexity. VideoMamba extends this success to the more challenging video domain, where the advantages of linear scaling become even more pronounced due to the multiplicative increase in token count from temporal frames.

The fundamental advantage of Mamba over attention-based models becomes particularly stark in the video domain. Consider a video of 16 frames at 224x224 resolution with patch size 16: this produces approximately 3,136 tokens. Self-attention requires ~9.8 million pairwise computations. Scaling to 64 frames produces 12,544 tokens and ~157 million attention computations — a 16x increase for only 4x more frames. In contrast, VideoMamba's SSM processes the same 64-frame input with only 4x the compute of 16 frames, maintaining perfect linear scaling. This makes VideoMamba uniquely suited for applications requiring long video understanding, such as movie analysis, surveillance, and instructional video comprehension.

VideoMamba extends the bidirectional Mamba (Vim) architecture from images to videos. We treat a video as a sequence of 3D patches (spatial-temporal tubes) and process them through stacked bidirectional SSM blocks. Each block applies the selective scan mechanism in both forward and backward directions along the flattened patch sequence, enabling the model to capture dependencies from both temporal directions. To address the challenge of training data scarcity for video models, we introduce a self-distillation strategy that transfers knowledge from a pretrained image Mamba model to initialize the video model, enabling effective training even with limited video data.

The selective scan mechanism in Mamba is key to its efficiency and expressiveness. Unlike traditional SSMs with fixed dynamics, Mamba uses input-dependent state transitions: the matrices governing state evolution are computed as functions of each input token. This makes the model content-aware while maintaining linear complexity. For video, this means the model can selectively attend to informative frames while efficiently skipping redundant ones. The bidirectional scanning ensures that temporal context from both past and future is available for each token's representation. We use a spatial-first, temporal-second flattening order that preserves spatial locality within each frame before connecting across time.

The self-distillation strategy addresses a practical bottleneck: large-scale video datasets are scarce compared to image datasets. We initialize VideoMamba from a pretrained Vim (Vision Mamba) model trained on ImageNet. The key challenge is adapting the 2D patch embedding to 3D: we inflate the 2D patch embedding weights by repeating them along the temporal dimension and dividing by the number of temporal frames, ensuring the output magnitude remains consistent. During training, we use a distillation loss that encourages the video model to maintain the spatial understanding of the image model while learning temporal dynamics. This allows VideoMamba to achieve strong performance with only ImageNet-1K pretraining, without requiring large-scale video pretraining datasets like Kinetics-710 or HowTo100M.

The VideoMamba architecture comes in three sizes to accommodate different computational budgets: VideoMamba-Ti (Tiny, 7M parameters), VideoMamba-S (Small, 26M), and VideoMamba-M (Middle, 74M). All variants use a patch size of 16x16x2 (spatial x temporal), processing videos at 224x224 spatial resolution. The number of bidirectional SSM blocks varies from 24 for Tiny to 32 for Middle, with hidden dimensions scaling accordingly. Position embeddings use learnable absolute embeddings that are interpolated when the number of input frames changes between training and inference, enabling flexible temporal resolution at test time.

VideoMamba demonstrates strong results across multiple video understanding benchmarks. On Kinetics-400, VideoMamba-M achieves 82.0% top-1 accuracy, competitive with Video Swin Transformer while being significantly more efficient (3x fewer GFLOPs). On the long-form video benchmark Breakfast (average 2.3 minutes), VideoMamba achieves 89.4% accuracy, surpassing TimeSformer by 4.7%, demonstrating its strength in long-term modeling. For multi-modal video-text retrieval on MSR-VTT, VideoMamba achieves 46.2 R@1, showing strong compatibility with language models. The self-distillation strategy provides 3.2% accuracy improvement on Kinetics-400 compared to training from scratch.

Efficiency analysis reveals VideoMamba's computational advantages. At 16 frames input, VideoMamba-M uses 47 GFLOPs compared to Video Swin-B's 282 GFLOPs and TimeSformer-L's 590 GFLOPs. More importantly, scaling to 64 frames increases VideoMamba's cost to only 188 GFLOPs (linear scaling), while Video Swin would require over 1100 GFLOPs (quadratic scaling). Throughput measurements on a single A100 GPU show that VideoMamba processes 38 videos/second at 16 frames, compared to 12 for Video Swin and 6 for TimeSformer. Ablation on the self-distillation reveals that spatial knowledge preservation accounts for 2.1% of the total 3.2% improvement, while the remaining 1.1% comes from the weight initialization itself.

We provide detailed analysis of where VideoMamba's advantages are most pronounced. On Something-Something V2, a dataset requiring fine-grained temporal reasoning (distinguishing "putting X into Y" from "taking X out of Y"), VideoMamba-M achieves 70.8% top-1 accuracy, outperforming Video Swin-T by 1.2% despite having fewer parameters. This demonstrates the selective scan's ability to focus on temporally discriminative moments. On the Epic-Kitchens-100 action anticipation task, requiring prediction of future actions from video context, VideoMamba achieves state-of-the-art results with 14.3% top-5 recall, suggesting that the SSM's sequential state propagation naturally suits temporal prediction tasks. Failure cases concentrate on activities requiring understanding of spatial relationships between multiple objects, where the flattened 1D sequence may lose critical spatial structure.

We have presented VideoMamba, demonstrating that State Space Models provide a scalable and efficient solution for comprehensive video understanding. Through bidirectional scanning, 3D patch tokenization, and self-distillation, VideoMamba achieves competitive or superior performance across short-term, long-term, and multi-modal video tasks while maintaining linear computational complexity. Our work establishes Mamba as a viable and often preferable alternative to Transformers for video understanding.

Future work should address several open questions. The flattening of 3D structure into 1D sequences may lose important spatial topology information; exploring multi-directional scanning strategies (e.g., separate spatial and temporal scans) could preserve this structure better. Combining VideoMamba with large language models for video question answering and video captioning represents a natural extension given the demonstrated multi-modal compatibility. The theoretical understanding of why SSMs work well for visual tasks remains incomplete and warrants further investigation, particularly regarding the relationship between the learned state dynamics and the visual features they encode.

The significance of VideoMamba extends beyond benchmark numbers to the broader question of whether attention is the only path to effective visual understanding. For the past five years, the field has witnessed a steady march toward Transformer dominance in vision, with ConvNets gradually being displaced. VideoMamba suggests that state space models represent a viable third paradigm — one that combines the global modeling capability of Transformers with the computational efficiency closer to ConvNets. For video understanding specifically, where sequence lengths easily reach tens of thousands of tokens, the SSM paradigm may ultimately prove more practical than attention-based approaches. The ongoing development of hardware-optimized SSM implementations further strengthens this case, as dedicated kernels close the gap between theoretical and actual computational savings.