This paper presents X3D, a family of efficient video networks that progressively expand a tiny 2D image classification architecture along multiple network axes — temporal duration, frame rate, spatial resolution, width, bottleneck width, and depth. Inspired by machine-driven architecture search in image recognition, X3D achieves "state-of-the-art performance while requiring 4.8x and 5.5x fewer multiply-adds and parameters" compared to previous methods.

Video understanding models have grown increasingly complex, with architectures like I3D, SlowFast, and Non-local Networks achieving strong accuracy but at significant computational cost. The design of these architectures largely relies on manual exploration guided by human intuition, raising the question: can we systematically find efficient video architectures through principled expansion from a minimal base model?

Drawing inspiration from EfficientNet's compound scaling in image recognition, X3D extends this principle to the video domain by considering six expansion axes: temporal duration, frame rate, spatial resolution, network width, bottleneck ratio, and network depth. Rather than scaling all axes simultaneously, X3D expands one axis at a time in a stepwise forward expansion, followed by backward contraction to meet target complexity.

Early approaches like C3D and I3D directly extend 2D convolutions to 3D, incurring heavy computational overhead. Separable convolutions (e.g., R(2+1)D, S3D) decompose 3D convolutions into spatial and temporal components for improved efficiency. SlowFast Networks introduce dual-pathway architectures operating at different temporal resolutions. Neural Architecture Search (NAS) has been applied to video models but typically searches within a fixed computational budget rather than across multiple complexity levels.

X3D begins with a minimal base architecture X2D, a lightweight 2D MobileNet-like model operating on a single frame. The forward expansion proceeds in steps: at each step, one axis is expanded by a fixed factor, trained and evaluated, and the axis yielding the best accuracy-complexity trade-off is selected. This greedy search explores only ~30 models total rather than the combinatorial space of all axis combinations.

After forward expansion reaches a desired accuracy level, backward contraction systematically reduces each axis to find the most efficient architecture at each target complexity. This produces a family of models: X3D-XS, X3D-S, X3D-M, X3D-L, X3D-XL, spanning a wide range of computational budgets from 0.6 GFLOPs to 48.4 GFLOPs. The expansion reveals that temporal resolution and spatial resolution are consistently the most impactful axes, while width expansion shows diminishing returns.

On Kinetics-400, X3D-M achieves 76.0% top-1 accuracy with only 6.2 GFLOPs, while SlowFast 8x8 R50 requires 65.7 GFLOPs for 77.0%. X3D-XL reaches 79.1% top-1 with 48.4 GFLOPs, comparable to SlowFast 16x8 R101+NL at 234 GFLOPs — achieving 4.8x fewer FLOPs and 5.5x fewer parameters. On Kinetics-600, X3D-XL achieves 81.9% top-1 accuracy. Transfer experiments on Charades and AVA confirm generalization ability.

X3D demonstrates that progressively expanding a tiny 2D architecture along multiple axes can produce a family of highly efficient video models. The stepwise expansion and contraction strategy achieves state-of-the-art accuracy-efficiency trade-offs with minimal search cost. The approach reveals that temporal and spatial resolution are the most critical expansion axes for video understanding, providing valuable design insights for future architectures.