We present DynamicFusion, the first dense SLAM system capable of reconstructing non-rigidly deforming scenes in real-time, using only a single commodity depth camera. We achieve this by estimating a dense volumetric 6D motion field (a warp field) that maps the live frame to a canonical model frame. We fuse the live depth maps into the canonical model by first estimating the warp field, then applying the inverse warp to integrate measurements. The warp field is parameterized as a set of deformation nodes with associated dual quaternion transformations, solved efficiently using a GPU-accelerated Gauss-Newton solver. Our system enables dense 3D reconstruction of complex, non-rigidly moving scenes such as people, animals, and clothing, tracking and accumulating surface detail over time at real-time frame rates (30 Hz).

Real-time 3D reconstruction from depth cameras has become a transformative technology. KinectFusion demonstrated that dense, high-quality 3D models can be built in real-time using a moving depth camera. However, KinectFusion and subsequent systems fundamentally assume that the scene is rigid — objects cannot move or deform. This is a severe limitation: most real-world scenes contain non-rigid objects such as people, animals, or deformable materials. Previous methods for non-rigid reconstruction require either pre-scanned templates, multi-view setups, or are limited to offline processing.

We present DynamicFusion, a system that extends dense SLAM to non-rigid scenes without templates, using a single depth camera in real-time. Our key insight is to represent the scene motion as a volumetric warp field that maps every point in the live frame to its corresponding location in a canonical model. By estimating this warp field per frame using dense non-rigid alignment, we can fuse incoming depth data into a single, cumulative volumetric model that grows in detail over time. The warp field is efficiently represented using a set of sparse deformation nodes with dual quaternion interpolation, enabling both real-time performance and smooth, physically plausible deformations.

Dense SLAM systems such as KinectFusion perform real-time reconstruction by fusing depth maps into a truncated signed distance function (TSDF) volume using the iterative closest point (ICP) algorithm for camera tracking. Extensions have improved scalability and accuracy, but all assume scene rigidity. Non-rigid surface reconstruction methods, including template-based approaches and physics-based simulation, typically require known geometry, multi-camera setups, or extensive offline computation. Embedded deformation graphs have been used for non-rigid registration in offline settings, and we adapt this representation for real-time dense tracking.

The warp field W maps every 3D point from the canonical model space to the live frame space. It is parameterized by a set of N deformation nodes, each with a position, a dual quaternion transformation, and a radial basis weight. The warp at any point is computed as a dual quaternion blending (DQB) of the K nearest deformation nodes, weighted by their radial basis function distances. Dual quaternion blending ensures smooth, artifact-free interpolation — unlike linear blend skinning, it avoids the "candy wrapper" artifacts associated with linear interpolation of rigid transformations. New deformation nodes are dynamically added as new surface geometry is discovered.

The fusion pipeline operates in three stages per frame. First, dense non-rigid ICP alignment estimates the warp field parameters by minimizing the point-to-plane error between the warped canonical model and the live depth map. This is solved using a GPU-parallelized Gauss-Newton optimization with a data term, regularization term (as-rigid-as-possible), and Tikhonov damping. Second, the estimated warp field is applied inversely to warp the live depth map into the canonical frame. Third, the warped data is integrated into the TSDF volume using standard volumetric fusion. This three-stage pipeline enables progressive accumulation of surface detail from a non-rigidly moving scene, just as KinectFusion does for rigid scenes.

We evaluate DynamicFusion on a variety of non-rigid scenes captured with a single Kinect sensor, including people performing actions, hands manipulating objects, and a dog. The system runs at approximately 30 Hz on a desktop GPU. Qualitative results show that the system successfully tracks and reconstructs complex deformations such as facial expressions, hand articulation, and body motion. The cumulative model progressively gains detail that is not visible in any single frame. We compare against rigid KinectFusion, which fails catastrophically when any part of the scene moves. Failure modes include very fast motions that exceed the convergence basin of the ICP solver, and topological changes such as surfaces separating.

We have presented DynamicFusion, the first system for real-time dense reconstruction of non-rigid scenes from a single depth camera. Our approach extends the volumetric fusion paradigm to handle arbitrary non-rigid motion through a dense, per-frame warp field. The dual quaternion-based deformation graph enables smooth, artifact-free warping while maintaining real-time performance. We believe DynamicFusion opens new possibilities for interactive applications including telepresence, augmented reality, and performance capture with commodity sensors.