We present a system for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes. Traditional structure-from-motion (SfM) and monocular SLAM techniques typically struggle with dynamic scenes lacking substantial parallax. We demonstrate that a deep visual SLAM framework, with careful modifications to its training and inference schemes, can scale to real-world videos of complex dynamic scenes with unconstrained camera paths. Our key contributions include integrating monocular depth priors and motion probability maps into differentiable SLAM, introducing uncertainty-aware global bundle adjustment, and obtaining consistent video depths without test-time network fine-tuning.

Extracting camera parameters and scene geometry from images represents a fundamental problem in computer vision, commonly referred to as structure from motion (SfM) or Simultaneous Localization and Mapping (SLAM). Conventional algorithms perform well with stationary scenes featuring large camera baselines but often falter when applied to casual monocular videos captured in uncontrolled settings. Casual videos typically exhibit limited camera motion parallax, broad focal length ranges, and moving objects — all of which violate the assumptions of classical SfM pipelines.

Recent approaches employ two main strategies: optimizing through mono-depth network fine-tuning or radiance field reconstruction, or combining intermediate estimates via global optimization. However, these approaches are computationally expensive or brittle when applied to unconstrained videos. Methods based on test-time optimization require hours of processing per video, while learning-based approaches like MonST3R adopt 3D point cloud representations and localize cameras via alignment optimization, but may lack robustness on diverse in-the-wild footage.

We introduce MegaSaM, providing accurate, fast and robust camera tracking and depth estimation from in-the-wild monocular videos. Our key innovations include: extending deep visual SLAM frameworks for dynamic content by predicting per-pixel motion probability maps; integrating monocular depth priors into differentiable bundle adjustment for initialization and regularization; introducing uncertainty-aware global bundle adjustment that leverages the approximate Hessian to determine when depth regularization is needed; and obtaining consistent video depths through first-order optimization without costly test-time network fine-tuning.

SLAM and SfM are used to estimate camera parameters and 3D scene structure from video sequences or unstructured image collections. Conventional approaches estimate 2D correspondences through feature matching or photometric alignment, then optimize 3D points and camera parameters via bundle adjustment. Recently, deep visual SLAM and SfM systems have emerged that adopt deep neural networks to estimate pairwise or long-term correspondences. While demonstrating accurate tracking, these methods typically assume predominantly static scenes and sufficient camera baselines between frames, degrading with scene dynamics or limited parallax.

Several recent works address dynamic scene challenges: Robust-CVD and CasualSAM jointly estimate camera parameters and dense depth maps from dynamic videos by optimizing spatially varying splines or fine-tuning monocular depth networks. Particle-SfM and LEAP-VO infer moving object masks from long-range trajectories, downweighting dynamic feature contributions. The concurrent work MonST3R adopts 3D point cloud representations and localizes cameras via alignment optimization. In the domain of monocular depth, recent models like DepthAnything show strong generalization on in-the-wild single images, but single-image models produce temporally inconsistent depth from videos.

Several works employ time-varying radiance field representations for dynamic scene reconstruction and novel view synthesis. These methods typically represent scenes as 4D spatiotemporal radiance fields that capture both geometry and appearance changes over time. The authors note that their work is "orthogonal to most of these techniques since most radiance field reconstruction methods require camera parameters or video depth maps as inputs." MegaSaM can thus serve as a preprocessing step that provides the camera poses and depths needed by downstream reconstruction methods.

Given an unconstrained video sequence, the goal is to estimate camera poses, focal length, and dense depth maps without constraints on camera or object motions. Deep visual SLAM systems maintain two state variables: per-frame low-resolution disparity maps and camera poses. These are iteratively updated through a differentiable bundle adjustment (DBA) layer. Given two video frames as input, the system predicts a 2D correspondence field and confidence through convolutional gated recurrent units iteratively. The rigid-motion correspondence field derives from camera ego-motion and disparity through multi-view constraints.

Since DROID-SLAM assumes known focal length, MegaSaM extends the formulation to jointly optimize camera poses, focal length, and disparity by minimizing weighted reprojection cost between network-predicted flows and rigid-motion flows. The Levenberg-Marquardt algorithm performs optimization in a fully differentiable manner. The approximate Hessian separates into block matrix form, with the disparity Hessian being diagonal, enabling efficient computation via the Schur complement trick. The flow and uncertainty predictions are trained end-to-end from a collection of synthetic video sequences of static scenes, with a combined loss of camera and flow supervision.

Deep Visual SLAM performs well for static scenes with substantial camera translation but its performance degrades when operating on videos of dynamic content, or videos with limited parallax. To address this, MegaSaM introduces motion probability prediction: an additional network predicts per-pixel object movement probability maps conditioned on frames and neighboring keyframes. These maps are supervised using multi-frame information to predict dynamic content pixels. A two-stage training scheme trains models on mixed static and dynamic videos: first, ego-motion pretraining on static synthetic data; second, dynamic fine-tuning that freezes the main network and fine-tunes only the motion module on synthetic dynamic videos, decorrelating scene dynamics learning from 2D correspondence learning.

While DROID-SLAM initializes disparity to constant values, this fails on videos with limited baselines and complex dynamics. MegaSaM instead initializes disparity with monocular depth priors, using DepthAnything with global scale and shift from ground truth during training. During inference, per-frame disparity maps initialize with metric-aligned monocular disparity combining DepthAnything relative disparity with UniDepth metric estimates through affine alignment. Training further initializes the first two camera poses to ground truth and perturbs focal length estimates by 25% to promote robustness.

The key innovation is uncertainty-aware global bundle adjustment. The critical question is whether to apply mono-depth regularization during global bundle adjustment: sufficient camera baseline makes regularization unnecessary, while rotational cameras with limited baseline require it to avoid degenerate solutions. MegaSaM explores the approximate Hessian matrix, using Laplace approximations to estimate posterior covariance through the inverse Hessian. Since full inversion is expensive, diagonal Hessian approximation estimates epistemic uncertainty. High uncertainty indicates unobservable parameters: rotational-dominant videos show much higher disparity uncertainty than forward-moving cameras. This uncertainty quantification measures observability, determining where mono-depth regularization applies and when focal length optimization should be disabled.

Optionally, obtaining more accurate and consistent higher-resolution video depth requires additional first-order optimization on video depths with per-frame aleatoric uncertainty maps. The objective comprises three cost functions: pairwise 2D flow reprojection loss for geometric consistency, temporal depth consistency loss encouraging coherence via optical flow, and scale-invariant mono-depth prior loss preventing excessive deviation from initial estimates. Key differences from CasualSAM include: performing direct optimization instead of costly mono-depth fine-tuning, fixing camera parameters rather than jointly optimizing cameras and depths, and adopting surface normal consistency and multi-scale depth gradient matching losses.

The depth prior loss comprises three components: scale-invariant depth loss computing mean square error between optimized and initial log-disparities; multi-scale gradient matching comparing log-disparity gradients to preserve local depth structures; and surface normal loss encouraging normal consistency derived from depth maps. The optimization conducts a 100-step warm-up phase optimizing only uncertainty maps, then jointly optimizes disparity and uncertainty maps for 400 steps. This separation enables the system to first calibrate per-pixel reliability estimates before using them to guide depth refinement.

Experiments evaluate MegaSaM on three benchmarks. MPI Sintel includes 18 animated sequences with complex object motions and camera paths, each containing 20-50 images. DyCheck contains real-world hand-held camera videos of dynamic scenes with 180-500 frames, using Shape of Motion's refined camera parameters and sensor depths as ground truth. An In-the-Wild benchmark includes 12 videos with long duration (100-600 frames), uncontrolled camera paths, and complex scene motions. Standard error metrics evaluate camera pose estimation: Absolute Translation Error (ATE), Relative Translation Error (RTE), and Relative Rotation Error (RRE). Video depth metrics include Absolute Relative Error, log RMSE, and Delta accuracy.

Results on three benchmarks demonstrate significant improvements. MegaSaM achieves best camera tracking accuracy on all error metrics in both calibrated and uncalibrated settings while maintaining competitive runtime. On Sintel, MegaSaM achieves ATE of 0.018 (calibrated) and 0.023 (uncalibrated); on DyCheck, ATE of 0.020 in both settings; and on in-the-wild footage, ATE of 0.004 in both settings. Depth estimates also significantly outperform baselines: Sintel abs-rel of 0.21 versus CasualSAM's 0.31 and DyCheck abs-rel of 0.11 versus CasualSAM's 0.21. The system outperforms concurrent work MonST3R in both robustness and accuracy.

Ablation studies validate each design choice. Key findings: mono-depth initialization significantly improves results over constant initialization; the two-stage training scheme proves critical for stable convergence; object movement map prediction substantially enhances performance on dynamic scenes; uncertainty-aware bundle adjustment prevents degenerate solutions on rotational-dominant videos; fixing camera poses during depth optimization outperforms joint refinement; and the new depth prior losses (surface normal and gradient matching) further improve quality. Visual comparisons show MegaSaM's estimated camera trajectories closest to ground truth, with more accurate, detailed, and temporally consistent depth maps.

The approach has several limitations. It fails in "extremely challenging scenarios": when moving objects dominate entire images or when there is nothing reliable to track. The system cannot handle varying focal lengths or strong radial distortion within a single video. Additionally, scenarios where camera motion and object motion are colinear — such as selfie videos — remain problematic. Incorporating "better priors from current vision foundation models" represents a promising direction for future work.

MegaSaM produces "accurate camera parameters and consistent depths from casual monocular videos of dynamic scenes," efficiently scaling "to in-the-wild footage of varying time duration, with unconstrained camera paths and complex scene dynamics." The work demonstrates that extending deep visual SLAM frameworks through careful modifications — including motion probability prediction, monocular depth integration, and uncertainty-aware bundle adjustment — achieves strong generalization and significantly outperforms recent state-of-the-art methods across diverse benchmarks. The system can serve as a reliable preprocessing module for downstream tasks such as dynamic scene reconstruction and novel view synthesis.