This paper presents RAFT-3D, a new deep architecture for 3D scene flow estimation. RAFT-3D iteratively updates a dense field of pixelwise SE(3) motion, achieving state-of-the-art performance by representing scene motion as a collection of rigid-body transforms. The key innovation is rigid-motion embeddings, which provide a soft grouping of pixels into rigid objects, enabling the network to reason about scene structure. The system operates on RGB-D video and outperforms prior methods on FlyingThings3D and real-world scenes.

Scene flow refers to the dense 3D motion field of a scene, capturing how every point moves in three-dimensional space between consecutive frames. Unlike optical flow, which only captures 2D pixel displacement, scene flow provides complete 3D motion information essential for autonomous driving, robotics, and dynamic scene understanding. Existing approaches typically decompose the problem into separate stereo matching and optical flow estimation, or operate directly on point clouds.

Current scene flow methods face significant limitations. Decomposition-based approaches suffer from error accumulation across sub-tasks and fail to exploit the geometric structure of rigid objects in the scene. Point cloud methods are limited by sparse and irregular point distributions, making it difficult to achieve dense, accurate predictions. Most critically, existing methods treat each pixel independently without considering that real-world scenes are composed of rigid bodies moving coherently.

RAFT-3D addresses these limitations by building upon the RAFT optical flow architecture and extending it to 3D. The method represents motion using dense SE(3) fields, where each pixel is assigned a rigid-body transformation. By introducing rigid-motion embeddings that softly group pixels sharing similar motion, the network can propagate information between pixels belonging to the same rigid object, enabling globally coherent motion estimation.

Early scene flow methods formulated the problem as variational energy minimization, jointly estimating depth and motion. More recent learning-based approaches like FlowNet3D and PointPWC-Net operate directly on 3D point clouds using learned feature matching. However, these methods still process points independently and lack explicit modeling of rigid-body motion structure.

The RAFT (Recurrent All-Pairs Field Transforms) architecture demonstrated that iterative refinement through a recurrent update operator on correlation volumes achieves superior optical flow estimation. Rigid-body segmentation methods have shown that scenes can be decomposed into independently moving rigid components, but these typically require separate segmentation and motion estimation stages.

RAFT-3D represents the motion of each pixel as a member of SE(3), the special Euclidean group of rigid-body transformations in 3D space. Each pixel is assigned a rotation (via unit quaternion) and a translation vector, forming a dense SE(3) field over the entire image. This representation inherently respects the geometric constraints of rigid motion, ensuring physically plausible scene flow estimates.

The core innovation is the rigid-motion embedding — a learned per-pixel feature vector that encodes which rigid body each pixel belongs to. Pixels with similar embeddings are softly grouped together, enabling information to flow between pixels belonging to the same rigid object during the iterative update process. This is implemented through a dense correlation volume weighted by embedding similarity, allowing the update operator to reason about rigid-body structure without explicit segmentation.

Following RAFT's design, RAFT-3D employs a GRU-based recurrent update operator that iteratively refines the SE(3) motion field. At each iteration, the operator looks up correlation features from a 4D correlation volume, combines them with context features and the current motion estimate, and produces an update to the SE(3) field. The rigid-motion embeddings modulate this process by enabling motion information to propagate across spatially distant but rigidly connected pixels, leading to faster convergence and more accurate results.

Experiments are conducted on FlyingThings3D and KITTI Scene Flow benchmarks. On FlyingThings3D, RAFT-3D achieves 3D end-point-error (EPE) of 0.051, reducing error by over 46% compared to the previous best method. On KITTI 2015 Scene Flow, RAFT-3D obtains state-of-the-art results across all metrics including SF-all (5.77%). Ablation studies confirm that rigid-motion embeddings contribute a 25% reduction in 3D EPE compared to a baseline without embeddings. The method also demonstrates strong generalization: models trained on synthetic data transfer effectively to real-world driving scenes.

RAFT-3D presents a new approach to scene flow estimation by operating in the space of rigid-body motions (SE(3)) and introducing rigid-motion embeddings for soft pixel grouping. The iterative update framework, inherited from RAFT, combined with structure-aware motion propagation through learned embeddings, achieves state-of-the-art performance on both synthetic and real-world benchmarks. The work demonstrates that incorporating geometric priors about rigid-body motion into deep networks leads to more accurate and physically plausible scene flow estimation.