Convolutional neural networks (CNNs) have recently been very successful in a variety of computer vision tasks, especially those linked to recognition. Optical flow estimation has not been among the tasks where CNNs were successful. In this paper, we propose two CNN architectures for optical flow estimation: a generic architecture (FlowNetSimple) and one including a correlation layer (FlowNetCorr). We also introduce the Flying Chairs synthetic dataset with 22,872 image pairs and demonstrate that networks trained on unrealistic synthetic data generalize surprisingly well to realistic datasets like Sintel and KITTI, achieving competitive real-time performance at 5-10 fps.

Optical flow estimation — predicting the per-pixel motion between two consecutive frames — is a fundamental problem that differs from recognition tasks in requiring precise per-pixel localization while also finding correspondences between two images. Traditional approaches rely on hand-crafted energy functions with smoothness constraints (e.g., Horn-Schunck, Lucas-Kanade), which are computationally expensive and struggle with large displacements and occlusions. The key challenge for applying CNNs to optical flow is the lack of large-scale ground truth training data: existing datasets like Sintel have only hundreds of training pairs, far too few for CNN training.

We propose two architectures. FlowNetSimple simply stacks both input images and processes them through a generic network with nine convolutional layers, relying on the network to implicitly learn matching. FlowNetCorr uses two separate processing streams with a novel correlation layer that performs multiplicative patch comparisons between feature maps from the two streams. The correlation of two patches centered at positions x1 and x2 is defined as the scalar product of the two vectors containing the values of the cropped patches. Both architectures include refinement through "upconvolutional" layers that progressively restore spatial resolution, with optional variational refinement for further smoothing.

Since existing ground truth datasets are far too small for CNN training, we create the Flying Chairs dataset by overlaying 809 chair models from rendered 3D models onto 964 Flickr background images. The process randomly samples affine transformations (rotation, translation, scaling) to create 22,872 image pairs with dense ground truth flow fields. Displacement distributions are matched to the Sintel dataset. Data augmentation during training includes geometric transformations (translation ±20%, rotation ±17°, scaling 0.9-2.0), Gaussian noise, and brightness/contrast/gamma variations. Despite its obviously unrealistic nature — chairs floating against random backgrounds — this dataset proves remarkably effective for training generalizable optical flow networks.

Training uses the Adam optimizer (β1=0.9, β2=0.999) with learning rate 1e-4, halved every 100k iterations after 300k. On Sintel Clean, FlowNetS achieves 4.50 EPE; on Sintel Final, 7.42 EPE. On KITTI, FlowNetS reaches 8.26 EPE; on Flying Chairs, 2.71 EPE. Networks trained only on Flying Chairs beat established methods like LDOF without any fine-tuning. FlowNetS generalizes better to Sintel Final; FlowNetC excels on Flying Chairs but struggles with large displacements. GPU processing achieves 0.08-0.15 seconds per frame, orders of magnitude faster than CPU-based competitors. FlowNetC slightly more overfits to the training data despite having similar parameter counts, and the correlation layer's limited displacement range explains performance gaps on datasets with large motions.

We have demonstrated that CNNs can learn optical flow through end-to-end training. Remarkably, networks trained on unrealistic synthetic data achieve competitive results on real-world datasets without fine-tuning. The FlowNetSimple and FlowNetCorr architectures offer different trade-offs between generalization and specialization. The generalization capabilities of the presented networks suggest that future improvements will come as more realistic training data becomes available and as architectures are further refined.