We present a test-time optimization method for estimating dense and long-range motion from video. Our approach, OmniMotion, represents a video using a quasi-3D canonical volume and maps between this volume and each frame through learned bijective transformations. This representation enables computing accurate, full-length motion trajectories for every pixel while tracking through occlusions and maintaining global consistency. We demonstrate that OmniMotion significantly outperforms prior state-of-the-art methods on the TAP-Vid benchmark.

Estimating dense pixel correspondences across video frames is fundamental to computer vision. Three key challenges persist: maintaining accurate tracks across long sequences, tracking points through occlusions, and maintaining coherence in space and time. Traditional methods use either sparse feature tracking — effective for distinctive points but limited in coverage — or dense optical flow, which works well for consecutive frames but accumulates drift errors when chained over long sequences. We propose a holistic approach that uses "all the information in a video to jointly estimate full-length motion trajectories".

Prior approaches to long-range motion estimation include flow chaining (concatenating pairwise optical flow), PIPs (tracking within 8-frame windows), and TAP-Net (single-frame predictions). Flow chaining methods like RAFT produce excellent pairwise flow but accumulate errors over long sequences and lack temporal coherence. PIPs handles occlusions within temporal windows but requires chaining for longer sequences. TAP-Net makes independent per-frame predictions that lack temporal context. Deformable Sprites provides a layered representation but is limited to fixed layer ordering. Our work fundamentally differs by optimizing a global 3D representation that enforces consistency across the entire video.

OmniMotion represents the entire video using a canonical 3D volume — a coordinate-based network F_theta that maps each canonical 3D coordinate to density (sigma) and color (c). This serves as a three-dimensional atlas of the observed scene, similar to a NeRF but designed not for novel-view synthesis but for establishing dense correspondences across time. The key insight is that by mapping all frames to a shared 3D space, correspondences become implicit — any two points in different frames that map to the same canonical location are in correspondence.

The representation defines invertible mappings T_i between local frame coordinates L_i and canonical space: u = T_i(x_i). Crucially, "bijective mappings ensure that the resulting correspondences between 3D points in individual frames are all cycle consistent". The bijections are parameterized using Real-NVP (invertible neural networks) conditioned on per-frame latent codes psi_i. To map between frames i and j: x_j = T_j^(-1) compose T_i(x_i). The method creates a "quasi-3D representation" as a "relaxation of dynamic multi-view geometry", allowing flexible modeling without solving the ill-posed full 3D reconstruction problem.

For any query pixel p_i in frame i, the motion computation proceeds as: (1) lift to 3D by sampling K points along a ray, (2) map to canonical space via T_i to obtain densities and colors, (3) map to target frame j via T_j^(-1), (4) aggregate via alpha compositing using density-based weights, and (5) project to 2D to obtain the predicted location. The optimization uses three complementary losses: flow loss (MAE against input optical flow), photometric loss (MSE against observed colors), and regularization loss (penalizing large accelerations in 3D trajectories).

We evaluate on the TAP-Vid benchmark comprising DAVIS (30 real videos), Kinetics (100 videos), and RGB-Stacking (50 synthetic videos). On DAVIS, OmniMotion achieves AJ: 51.7 (best), position accuracy: 67.5 (best), occlusion accuracy: 85.3 (best), and temporal coherence: 0.74 (best). The improvements are substantial: compared to the strongest baseline, AJ improves by 1.1 points absolute while temporal coherence improves dramatically. Ablation studies confirm all components are critical: removing invertible networks drops AJ from 51.7 to 12.5; removing photometric loss drops AJ to 42.3; uniform sampling instead of hard mining drops AJ to 47.8.

OmniMotion presents a complete solution to dense, long-range video motion estimation. Through its quasi-3D canonical representation with guaranteed cycle consistency via bijective mappings, the method achieves state-of-the-art performance while handling general videos with arbitrary camera and scene motion. The test-time optimization approach leverages global information across entire videos, fundamentally addressing limitations of local, frame-pair-based methods. Limitations include struggles with rapid non-rigid motion, sensitivity to initialization, and quadratic scaling with sequence length.