We present DUSt3R, a new paradigm for Dense and Unconstrained Stereo 3D Reconstruction from arbitrary image collections. Our approach does not require prior information about camera calibration or viewpoint poses. We cast the pairwise reconstruction problem as a regression of pointmaps, relaxing the hard constraints of usual projective camera models. We show that this formulation smoothly unifies the monocular and binocular reconstruction cases. In the multi-view case, we introduce a simple yet effective global alignment procedure. We demonstrate that our formulation allows leveraging powerful pretrained models and leads to state-of-the-art results on several 3D vision benchmarks, including monocular and multi-view depth estimation, relative and absolute pose estimation, and 3D reconstruction.

Our key insight is that by directly regressing 3D pointmaps from image pairs, we can bypass the traditional sequential pipeline of keypoint detection, matching, essential matrix estimation, and triangulation. Each of these steps can introduce errors that propagate downstream, leading to brittle systems that fail with few views, non-Lambertian surfaces, or insufficient camera baselines. DUSt3R replaces this fragile pipeline with a single feed-forward network that learns geometric priors from large-scale training data.

Reconstructing the 3D geometry and camera parameters from uncalibrated image collections is a fundamental challenge in computer vision. Traditional Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines decompose the problem into sequential steps — keypoint matching, essential matrix estimation, triangulation, bundle adjustment — each introducing potential errors that propagate downstream. This sequential approach is unsatisfactory because error accumulation occurs across stages and communication between substeps is absent. Specific failures include SfM breakdown with few views, non-Lambertian surfaces, and insufficient camera motion.

DUSt3R takes a radically different stance by operating without any camera calibration requirements. The core innovation involves regressing pointmaps — dense 2D fields of 3D points — directly from image pairs, eliminating explicit geometric constraints while leveraging learned priors from large datasets. This formulation naturally unifies monocular and multi-view scenarios: a single network handles both cases by outputting two pointmaps in a shared coordinate frame, with the network architecture permitting leverage of powerful pretrained models like CroCo. A simple global alignment procedure extends the pairwise approach to handle multi-image scenarios without traditional bundle adjustment complications.

A pointmap is formally defined as X in R^(W x H x 3), establishing a one-to-one correspondence between image pixels and 3D scene points such that I_(i,j) corresponds to X_(i,j) for all pixel coordinates. The network processes two RGB images and outputs two pointmaps expressed in the first image's coordinate frame, plus associated confidence maps. This design choice — outputting both pointmaps in a shared coordinate system — represents a key departure from conventional approaches, as it implicitly embeds relative pose information without requiring explicit pose parameterization.

The architecture builds on CroCo and comprises: a Siamese ViT-Large encoder processing both images with shared weights, Transformer decoders with cross-attention mechanisms enabling constant information exchange between branches, and separate DPT regression heads outputting pointmaps and confidence maps. The cross-attention structure ensures properly aligned pointmap outputs by having each decoder block attend to tokens from both views sequentially. Importantly, the architecture enforces no explicit geometric constraints — instead, the network learns geometric priors purely from training data containing only geometrically consistent pointmaps.

The primary training objective uses Euclidean distance in 3D space rather than 2D image-space metrics. To handle scale ambiguity, predictions and ground-truth are normalized by average point distances from origin. The network jointly learns pixel-wise confidence scores indicating prediction reliability through a confidence-aware loss that includes a regularization term, forcing the model to extrapolate in difficult areas such as single-view regions and translucent surfaces. For multi-image scenarios, a global alignment procedure constructs a connectivity graph of image pairs and optimizes all pairwise predictions into a joint 3D space through rigid transformations and per-pair scaling factors, converging within hundreds of gradient descent iterations.

The model trains on 8.5 million image pairs from eight diverse datasets: Habitat, MegaDepth, ARKitScenes, Static Scenes 3D, Blended MVS, ScanNet++, CO3D-v2, and Waymo — covering indoor, outdoor, synthetic, real-world, and object-centric scenarios. The architecture employs a ViT-Large encoder and ViT-Base decoder with DPT head, initialized from CroCo pretraining. Training proceeds sequentially: first at 224x224 resolution, then at 512-pixel maximum dimension with variable aspect ratios per batch for generalization.

On multi-view pose estimation using CO3Dv2 and RealEstate10K datasets, DUSt3R with global alignment achieves 96.2% RRA@15 and 86.8% RTA@15 on CO3Dv2, significantly surpassing the state-of-the-art PoseDiffusion. Both PnP and global alignment variants outperform existing methods, demonstrating strong performance even with limited input views (3-10 frames). On monocular depth estimation, the model outperforms self-supervised baselines and performs comparably to state-of-the-art supervised methods in a zero-shot transfer setting on DDAD, KITTI, NYUv2, BONN, and TUM datasets. For multi-view depth estimation on DTU, ETH3D, Tanks and Temples, and ScanNet, DUSt3R achieves state-of-the-art accuracy on ETH-3D, even outperforming methods that require ground-truth poses.

Systematic ablations demonstrate the critical importance of key design choices. CroCo pretraining provides consistent improvements across all tasks, validating the value of cross-view completion as a pretraining objective. Higher input resolution (512 pixels versus 224) yields substantial performance gains, suggesting that spatial detail is crucial for geometric reasoning. On 3D reconstruction quality using the DTU dataset in a zero-shot setting without finetuning, DUSt3R achieves 2.7mm average accuracy, 0.8mm completeness, and 1.7mm overall error, demonstrating practical reconstruction precision despite not matching pixel-accurate triangulation methods.

We have presented DUSt3R, a unified paradigm that solves multiple 3D vision tasks — reconstruction, depth estimation, pose estimation, and localization — without requiring camera calibration. By reformulating geometric problems as pointmap regression with learned priors, our work simplifies traditional pipelines while achieving competitive or superior performance across diverse benchmarks. The pointmap representation preserves pixel-to-point correspondence while handling implicit pose relationships. Our generic Transformer-based architecture leverages pretrained models without task-specific constraints, and the 3D global alignment optimization enables fast convergence for multi-view scenarios. DUSt3R demonstrates that relaxing explicit geometric constraints while leveraging large-scale pretraining and learned priors can effectively solve challenging unconstrained 3D reconstruction problems.