We present a real-time monocular dense SLAM system designed bottom-up from MASt3R, a two-view 3D reconstruction and matching prior. Our system demonstrates robustness on uncontrolled video sequences without assuming fixed or parametric camera models beyond a unique camera centre. We introduce efficient approaches for pointmap matching, camera tracking, local fusion, graph construction, loop closure, and second-order global optimisation. With known calibration, our system achieves state-of-the-art performance across various benchmarks, operating at 15 FPS while producing globally consistent poses and dense geometry.

Visual simultaneous localisation and mapping (SLAM) represents foundational technology for robotics and augmented reality applications. Despite advances in integrated hardware-software solutions, SLAM has not achieved plug-and-play status due to hardware expertise and calibration requirements. Even minimal monocular setups without additional sensors lack reliable solutions delivering both accurate poses and consistent dense maps in uncontrolled environments.

Dense monocular SLAM requires reasoning across time-varying poses, camera models, and 3D scene geometry. Various handcrafted and data-driven priors have been proposed to address this high-dimensional inverse problem. Single-view geometry priors, including monocular depth and surface normals, suffer from ambiguities and cross-view inconsistencies. Multi-view approaches like optical flow reduce ambiguity but struggle with entangled pose and geometry estimation, since pixel motion depends on both camera extrinsics and camera model parameters. The 3D scene structure remains invariant across viewpoints, providing the unifying prior needed for jointly solving poses, camera models, and dense geometry.

Two-view 3D reconstruction priors, pioneered by DUSt3R and its successor MASt3R, have transformed structure-from-motion by leveraging curated 3D datasets. These networks output pointmaps directly from two images in a common coordinate frame, such that the aforementioned subproblems are implicitly solved in a joint framework. The two-view architecture mirrors classical two-view geometry as the building block of SfM, enabling efficient decision-making and robust consensus in backend processing. We propose the first real-time SLAM framework leveraging two-view 3D reconstruction priors as foundational infrastructure for tracking, mapping, and relocalisation. Previous applications used these priors offline with unordered image collections, while SLAM processes data incrementally requiring real-time operation. This necessitates new approaches for low-latency matching, map maintenance, and efficient large-scale optimisation.

Sparse monocular SLAM focuses on jointly solving camera poses and selected unbiased 3D landmarks. Algorithmic advances exploiting optimisation sparsity and careful graph construction enabled real-time pose estimation and sparse reconstructions at large scales. While sparse monocular SLAM achieves accuracy with sufficient features and parallax, it lacks dense scene models useful for robust tracking and explicit geometry reasoning.

Early dense monocular SLAM systems demonstrated alternating optimisation of poses and dense depth with handcrafted regularisation in controlled settings. Recent work attempts combining data-driven priors with backend optimisation. Single-image geometric predictions including depth and surface normals show significant progress, though their SLAM applications remain limited. "Predicting geometry from a single-view is ambiguous, resulting in biased and inconsistent 3D geometry." SLAM literature has focused on predicting hypothesis spaces through latent spaces, subspaces, local primitives, and distributions, though robust multi-view correspondence remains essential.

DUSt3R introduced novel two-view 3D reconstruction providing dense 3D point clouds in common coordinate frames. Its successor MASt3R predicts additional per-pixel features improving pixel matching for localisation and SfM. "As with all priors, predictions can still have inconsistencies and correlated errors in the 3D geometry." DUSt3R and MASt3R-SfM require large-scale optimisation for global consistency, but time complexity scales poorly with image count. Spann3R forgoes backend optimisation by fine-tuning DUSt3R for direct pointmap streaming into global coordinates, but limited memory maintenance causes drift in larger scenes.

DUSt3R takes in a pair of images I_i, I_j and outputs pointmaps X_i^i, X_i^j along with their confidences C_i^i, C_i^j. In MASt3R, an additional head is added to predict d-dimensional features for matching D_i^i, D_i^j and corresponding confidences. Scale is often a large source of inconsistency across predictions, so all poses are defined as T in Sim(3) with updates using Lie algebra. "Our only assumption on the camera model is that of a generic central camera, which means that all rays pass through a unique camera centre." The function psi(X_i^i) normalises a pointmap into rays of unit norm such that each pointmap defines its own camera model.

"Correspondence is a fundamental component of SLAM that is required for both tracking and mapping." Given two images, we need to find pixel matches m_{i,j} = M(X_i^i, X_i^j, D_i^i, D_i^j). Naive brute-force matching has quadratic complexity since it is a global search over all possible pairs of pixels. DUSt3R uses a k-d tree approach, but "construction is non-trivial to parallelise and the nearest-neighbour search in 3D will find many inaccurate matches if there are errors in the pointmap predictions."

Drawing inspiration from optimisation as a local search, the authors propose iterative projective matching: given the two pointmap predictions from MASt3R, the reference pointmap is normalised psi(X_i^i) to give a smooth pixel-to-ray mapping. The core matching equation minimises angular error between rays: p* = arg min ||psi([X_i^i]_p) - psi(x)||^2. Minimising the Euclidean distance between normalised vectors is equivalent to minimising the angle between two normalised rays. The system solves iteratively using Levenberg-Marquardt and converges for almost all valid pixels within 10 iterations as the ray image is smooth. The implementation uses custom CUDA kernels, taking only 2 milliseconds for tracking and a few milliseconds for all newly added edges.

After geometric matching, MASt3R demonstrates that leveraging per-pixel features greatly improves downstream performance on pose estimation. The authors conduct a coarse-to-fine image-based search by updating the pixel location to the maximum feature similarity in a local patch window. To handle occlusions and outliers, matches with large distances in 3D space are invalidated. Critically, "matches are unbiased by pose estimates as they rely purely on the MASt3R outputs," which is atypical for projective data association.

The system estimates the relative transformation T_k^f between the current frame and the last keyframe. A straightforward approach would minimise 3D point error, but "point error is easily skewed by errors in the pointmap predictions as inconsistent predictions in depth are relatively frequent." Since information fuses into a single pointmap that averages out all predictions, error in tracking degrades the keyframe's pointmap that will also be used in the backend. Instead, the authors propose using directional ray error, which is less sensitive to incorrect depth predictions: E_r = sum ||psi(X_k^k) - psi(T_k^f * X_f^f) / w(q, sigma_r^2)||.

The system solves using Gauss-Newton in an iteratively reweighted least-squares (IRLS) framework, computing analytical Jacobians with respect to a perturbation of the relative pose. After solving for relative pose, the canonical pointmap is updated via a running weighted average filter. "The pointmap initially has larger errors and less confidence due to only having small baseline frames, but filtering is able to merge information from many viewpoints." Compared to MASt3R-SfM's canonical pointmap computation, this work computes this incrementally and requires transformation of the points since an additional network prediction would slow down tracking. This approach leverages information from all frames without having to explicitly optimise for all camera poses.

A new keyframe is added if the number of valid matches or unique keyframe pixels falls below a threshold. Upon addition, a bidirectional edge to the previous keyframe is added to the edge-list, constraining sequential poses. However, drift can still occur. To address this, the system adapts the Aggregated Selective Match Kernel (ASMK) framework used by MASt3R-SfM for image retrieval, modified to work incrementally. The database is queried with encoded features to obtain top-K images. If retrieval scores exceed a threshold, these pairs are given to the MASt3R decoder and bidirectional edges are added if sufficient matches are found. This enables loop closure detection and correction without relying on traditional feature descriptors like BoW or NetVLAD.

Given current estimates of keyframe poses T_{WC_i} and canonical pointmaps X_i^i, the goal of backend optimisation is to achieve global consistency across all poses and geometry. The authors introduce an efficient second-order optimisation scheme that handles the gauge freedom of the problem by fixing the first 7-DoF Sim(3) pose. They jointly minimise the ray error for all edges in the graph. Given N keyframes, the equation forms and accumulates 14x14 blocks into the 7Nx7N Hessian, solved using Gauss-Newton with sparse Cholesky decomposition.

Construction of the Hessian is made efficient through the use of analytical Jacobians and parallel reductions all implemented in CUDA. A small error term on distance consistency is added to avoid degeneracy in the pure-rotation case. At most 10 iterations of Gauss-Newton are performed for every new keyframe and optimisation terminates early upon convergence. "The use of second-order information greatly speeds up the global optimisation over the alternatives, and our efficient implementation ensures that it is not the bottleneck in the overall system."

If the system loses tracking due to an insufficient number of matches, relocalisation mode is triggered. New frames query the retrieval database with a stricter threshold on the score. Once retrieved images have sufficient matches, a new keyframe is added and tracking resumes. For known calibration, two straightforward changes are applied: first, pointmaps are constrained to be backprojected along rays defined by the known camera model; second, residuals in optimisation are changed to pixel space rather than ray space. "In the future, any parametric camera model and its corresponding Jacobian could be used here," suggesting extensibility of the framework.

The system was evaluated on multiple real-world benchmarks. On TUM RGB-D, the calibrated system achieved state-of-the-art trajectory error. Many of the previously best-performing algorithms, such as DROID-SLAM, DPV-SLAM, and GO-SLAM, build on the foundational matching and end-to-end system proposed by DROID-SLAM. The uncalibrated system significantly outperformed a baseline using GeoCalib for intrinsic calibration. On 7-Scenes, the calibrated system outperformed both NICER-SLAM and DROID-SLAM. On EuRoC, the system reports 0.041m average ATE across 11 sequences. For ETH3D-SLAM, the method has "a longer tail in terms of robustness" resulting in best ATE and area-under-curve metrics.

The team evaluated geometry against DROID-SLAM and Spann3R on EuRoC Vicon room sequences and 7-Scenes. Metrics included RMSE accuracy, completion, and Chamfer distance. On 7-Scenes, the uncalibrated method achieved best performance in accuracy and Chamfer distance metrics. For EuRoC, despite larger ATE, "our method outperforms DROID-SLAM in Chamfer distance," suggesting that the dense geometry produced by MASt3R-SLAM is more globally consistent than that of flow-based alternatives.

Ablation studies demonstrated that the ray error formulation for uncalibrated tracking and backend optimisation significantly improves performance over using the 3D point error. Regarding pointmap fusion methods, weighted fusion achieves the lowest ATE without calibration. The parallel projective matching achieved best accuracy with significantly faster runtime, taking 2ms versus 2 seconds for MASt3R matching. Qualitative results include reconstruction examples from the challenging Burghers sequence, TUM sequences, EuRoC data, and an example "with extreme zoom changes shown by two consecutive keyframes."

The authors acknowledge several limitations: they do not "currently refine all geometry in the full global optimisation" — while DROID-SLAM optimises per-pixel depth, their "framework permits incoherent geometry." Additionally, "MASt3R is only trained on pinhole images, so its geometry predictions degrade with increasing distortion." However, they anticipate future models will support diverse camera models compatible with their framework. Regarding efficiency, "using the decoder at full resolution is currently a bottleneck, especially for low-latency tracking and checking loop closure candidates."

The authors present "a real-time dense SLAM system based on MASt3R that handles in-the-wild videos and achieves state-of-the-art performance." They contrast their approach with recent SLAM progress by noting that recent work "has followed the contributions of DROID-SLAM, which trains an end-to-end framework." Instead, their system builds "around an off-the-shelf geometric prior that achieves comparable pose estimation for the first time, while also providing consistent dense geometry." This suggests a paradigm shift: from end-to-end trained SLAM systems to modular frameworks built upon powerful foundation priors.