We address the problem of 3D scene flow estimation from stereo video sequences. Our approach represents the dynamic 3D scene as a collection of planar, rigidly moving, local segments. We jointly estimate the assignment of pixels to segments along with the 3D shape and rigid motion parameters of each plane. Our formulation combines occlusion-sensitive data terms with regularizers on shape, motion, and segmentation within a unified energy minimization framework. The optimization proceeds in two stages: first estimating shape and motion parameters by assigning moving plane proposals, then updating pixel-to-segment assignments while keeping plane parameters fixed. Our method achieves leading performance on the KITTI benchmark, outperforming competing 3D scene flow methods and yielding superior 2D optical flow estimates compared to dedicated optical flow techniques.

Scene flow — the dense 3D motion field of a scene — is the natural extension of optical flow from 2D to 3D. Recovering scene flow from stereo sequences provides rich geometric and dynamic information for applications in autonomous driving, robotics, and 3D reconstruction. Existing methods typically estimate scene flow by independently computing stereo depth and optical flow, then combining them post-hoc. This decoupled strategy fails to exploit the mutual constraints between geometry and motion, leading to suboptimal results especially in occluded regions. We propose a joint estimation framework based on the assumption that real-world scenes can be well-approximated by a collection of rigidly moving planar segments.

Early scene flow methods by Vedula et al. (1999) required multiple calibrated cameras. Subsequent work by Huguet and Devernay (2007) formulated stereo scene flow as a variational optimization problem, but produced overly smooth results that fail to preserve motion boundaries. Piecewise planar models for stereo have been successful in static depth estimation; our work extends this to the dynamic setting with joint segmentation and rigid motion. In optical flow, recent piecewise rigid methods have shown that the rigidity assumption is surprisingly effective even for non-rigid scenes, as local regions often exhibit approximately rigid motion over short time intervals.

We represent the scene as a set of planar segments S = {s₁, ..., s_K}, where each segment s_k is characterized by a 3D plane (normal n_k and offset d_k) and a rigid motion (rotation R_k and translation t_k). Each pixel i is assigned to exactly one segment via a labeling function l: pixels → {1,...,K}. The depth of pixel i assigned to segment k is determined by the plane equation z_i = -d_k / (n_k · r_i), where r_i is the viewing ray. The 3D motion of each pixel is then given by applying the rigid transformation (R_k, t_k) to its 3D position. This representation compactly encodes both geometry and motion with only 9 parameters per segment.

The energy function consists of four terms: (1) a stereo data term measuring photometric consistency between left and right images at both time steps, with explicit occlusion reasoning; (2) a temporal data term measuring photometric consistency between consecutive frames; (3) a shape regularizer encouraging neighboring segments to have similar depth; and (4) a motion regularizer encouraging spatially adjacent segments to have consistent rigid motions. The explicit handling of occlusions in the data term is critical — occluded pixels are identified and excluded from the matching cost rather than being treated as outliers, leading to more accurate depth and motion estimates near object boundaries.

We evaluate on the KITTI Scene Flow benchmark, which provides ground truth from LiDAR measurements in real-world driving scenarios. Our method ranks 1st among all scene flow methods at the time of submission, with an average endpoint error of approximately 5.0 pixels for optical flow and 4.5% outliers for disparity estimation. Notably, our scene flow method also outperforms several dedicated optical flow methods on the 2D motion component, demonstrating that joint 3D reasoning improves even 2D motion estimation. Qualitative results show crisp motion boundaries and accurate depth discontinuities that are absent in variational methods. We further validate on synthetic sequences with known ground truth, achieving sub-pixel accuracy for both depth and flow.

We have presented a piecewise rigid scene flow method that jointly estimates 3D geometry, segmentation, and rigid motion from stereo video. The planar, rigidly-moving segment representation provides a compact and effective parameterization, while the occlusion-aware energy formulation enables accurate estimation near object boundaries. Our results on KITTI demonstrate state-of-the-art performance, and the finding that joint 3D scene flow estimation improves 2D optical flow highlights the value of holistic scene understanding. Future work may extend the framework to handle non-rigid deformations and exploit temporal consistency across longer sequences.