LongSplat addresses critical challenges in novel view synthesis (NVS) from casually captured long videos characterized by irregular camera motion, unknown camera poses, and expansive scenes. The method introduces three main components: Incremental Joint Optimization that concurrently optimizes camera poses and 3D Gaussians; a robust Pose Estimation Module leveraging learned 3D priors from MASt3R; and an efficient Octree Anchor Formation mechanism converting dense point clouds into anchors based on spatial density. The authors report achieving state-of-the-art results in rendering quality, pose accuracy, and computational efficiency.

High-quality 3D reconstruction and novel view synthesis are essential for applications such as virtual reality, augmented reality, virtual tourism, and cultural heritage preservation. Traditional methods rely on accurate poses from Structure-from-Motion (SfM) preprocessing, yet pipelines like COLMAP frequently fail in casual settings. The framework departs from conventional approaches by jointly optimizing camera poses and 3DGS in a unified framework, integrating correspondence-guided pose estimation with 3DGS geometry and photometric refinements.

Neural Radiance Fields (NeRF) revolutionized photorealistic rendering, with subsequent improvements in anti-aliasing, reflectance, sparse view training, and faster rendering. Recent 3D Gaussian Splatting "enables real-time rendering through explicit representations". However, "most approaches still rely on pre-computed camera parameters from SfM". For unposed NVS, methods like i-NeRF, BARF, and GARF "typically assume small pose perturbations, limited camera motion, or additional priors, struggling with challenging trajectories". Casual long videos present unique challenges: free-moving trajectories, lack of pose information, and continuously expanding scenes.

3D Gaussian Splatting represents scenes with explicit Gaussians, where each Gaussian is projected onto the image plane using the camera pose W. The anchor-based representation from Scaffold-GS divides scenes into sparse voxels, with k Gaussians initialized per anchor. Unlike Scaffold-GS's fixed-resolution approach, LongSplat constructs structured anchors from MASt3R's per-frame dense point clouds using an adaptive octree.

LongSplat constructs structured anchors from MASt3R's per-frame dense point clouds using an adaptive octree. The method progressively subdivides space based on local point density: voxels exceeding density threshold tau_split split into 8 smaller voxels, while low-density voxels are removed to reduce redundancy. Each anchor inherits a spatial scale proportional to its voxel size, ensuring coarse anchors for sparsely observed areas and finer anchors for detailed regions. This achieves 7.92x compression compared to dense point cloud representation.

For each new frame, MASt3R provides 2D correspondences, allowing back-projection of matched points to 3D. These correspondences are used to solve the initial pose via PnP, followed by photometric refinement. To correct MASt3R's depth scale drift, the method computes a scale factor by comparing rendered depth and MASt3R's aligned depth. Newly visible regions are detected via occlusion mask and unprojected into 3D, then converted into anchors via Octree Anchor Formation. If PnP fails, a fallback mechanism triggers global re-optimization of all past frames.

The process begins with a small set of initial frames using MASt3R for pose and point cloud estimation. After initialization, all 3D Gaussian parameters and camera poses are jointly optimized across all processed frames. Covisibility between frames is measured using Intersection-over-Union (IoU) of visible anchors, and frames with covisibility below threshold are excluded from the local optimization window. This visibility-adaptive mechanism ensures local Gaussians are consistently supervised by reliable multi-view constraints. The total loss combines photometric loss, depth loss from MASt3R's scale-aligned depth prior, and keypoint reprojection loss.

Evaluation is conducted on three datasets: Tanks and Temples (smooth forward-facing), Free Dataset (complex unconstrained), and Hike Dataset (long videos with complex geometry). On Tanks and Temples, LongSplat achieves state-of-the-art rendering quality with average PSNR 32.83 dB. Competing methods like CF-3DGS often face out-of-memory issues, while LocalRF produces fragmented geometry and pose drift. LongSplat achieves 281.71 FPS and trains in just 1 hour on an NVIDIA RTX 4090, nearly 30x faster than LocalRF.

Ablation studies confirm each component's contribution. Removing pose estimation, global optimization, or local optimization significantly degrades performance. The visibility-adapted window achieves the best balance compared to fixed-size windows. The adaptive octree achieves superior reconstruction quality with 7.92x memory compression. The method demonstrates robustness against pose drift, effectively minimizing drift and maintaining stable trajectories even over long sequences.

The paper presents a complete framework addressing casual long video reconstruction challenges. The method successfully integrates incremental joint optimization, a robust tracking module, and adaptive octree anchors, significantly improving pose accuracy, reconstruction quality, and memory efficiency. The authors note shared limitations with unposed reconstruction methods: assuming static scenes and fixed intrinsics, making it unsuitable for dynamic objects or varying focal lengths. Future work includes handling dynamic scenes and enhancing pose estimation robustness.