We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance on the Sintel and KITTI benchmarks. On Sintel (final pass), RAFT obtains an end-point-error of 2.855, a 16% error reduction from the best published result. RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count.

Optical flow is the task of estimating per-pixel motion between video frames. It is a fundamental problem in computer vision, with applications in action recognition, video editing, and autonomous navigation. Most recent approaches frame optical flow as a learning problem, directly regressing the flow field from a pair of images in a single forward pass. While these approaches have shown strong results, they lack the iterative refinement that is common in classical optimization-based methods, which use principles such as coarse-to-fine processing and variational inference.

We propose RAFT, a new architecture that combines the benefits of both paradigms. Our approach has three main components: (1) a feature encoder that extracts per-pixel features from both input images, (2) a correlation layer that computes visual similarity between all pairs of pixels to produce a 4D correlation volume, and (3) a recurrent update operator that iteratively refines the flow estimate by performing lookups on the correlation volume. Unlike prior work that operates at a single fixed resolution or uses a coarse-to-fine pyramid, RAFT maintains and updates a single high-resolution flow field.

Given a pair of consecutive images I_1 and I_2, we estimate a dense displacement field (f_1, f_2) which maps each pixel (u, v) in I_1 to its corresponding coordinates (u + f_1(u), v + f_2(v)) in I_2. Our approach produces a sequence of flow estimates {f_1, ..., f_N} that converges to a fixed point of the flow field. The feature encoder network g_theta maps both input images to dense feature maps at 1/8 resolution. We use a shared-weight architecture, applying the same network to both frames.

We compute the correlation volume by taking the dot product between all pairs of feature vectors. Given feature maps g_theta(I_1) in R^{H x W x D} and g_theta(I_2) in R^{H x W x D}, the correlation volume C(g_theta(I_1), g_theta(I_2)) in R^{H x W x H x W} is defined by computing the inner product for all pairs. We then construct a 4-layer correlation pyramid by repeatedly pooling the last two dimensions with kernel sizes 1, 2, 4, and 8. This provides access to both large displacements (coarser levels) and sub-pixel precision (finer levels) within a single framework.

The key operation in our approach is the correlation lookup. Given a current estimate of the flow field, we use the estimated correspondences to index the correlation volume and retrieve a local neighborhood of correlation values. Specifically, we define a local grid centered at the current flow estimate and use bilinear interpolation to sample the correlation volume at the grid points. The lookups are performed on all levels of the correlation pyramid, providing multi-resolution information while keeping the lookup cost constant.

The update operator is built using a gated recurrent unit (GRU). At each iteration, it takes as input the current flow estimate, the correlation features from the lookup operation, and the context features extracted from the first image. The GRU produces a flow update delta-f that is applied to the current estimate. This recurrent design allows the network to learn an optimization trajectory that mimics the iterative refinement of classical methods. We train the network by applying L1 loss on the sequence of flow predictions, with exponentially increasing weights, giving more importance to later (more accurate) predictions.

We evaluate RAFT on the Sintel and KITTI 2015 optical flow benchmarks. We first train on FlyingChairs followed by FlyingThings3D, then fine-tune on the target dataset. On Sintel (final pass), RAFT achieves an end-point-error (EPE) of 2.855, compared to 3.38 for the previous best method. On KITTI 2015, RAFT achieves 5.10% Fl-all, surpassing the previous state of the art at 6.24%. These results represent a 16% reduction in error on Sintel and an 18% reduction on KITTI.

We also evaluate the generalization ability of RAFT. When trained only on synthetic data (FlyingChairs + FlyingThings3D) and evaluated directly on Sintel and KITTI without fine-tuning, RAFT achieves significantly better results than prior methods that were also trained only on synthetic data. Additionally, we study the effect of the number of iterations at test time. RAFT shows consistent improvement as the number of iterations increases, with most of the gain captured in the first 12 iterations. The model uses only 5.3M parameters, fewer than most competing methods.

We have presented RAFT, a new architecture for optical flow that achieves state-of-the-art accuracy, strong generalization, and high efficiency. The key design elements are the all-pairs correlation volume, the multi-scale correlation pyramid, and the GRU-based iterative update operator. RAFT demonstrates that iterative refinement, when implemented through learned recurrent updates, can dramatically improve optical flow estimation. The simplicity and strong performance of RAFT suggest that this design philosophy — building dense correlation volumes and iteratively refining predictions — may be applicable to other dense correspondence problems such as stereo matching and scene flow estimation.