The authors present a CNN-based approach that predicts the future motion of each and every pixel in a static image in terms of optical flow. Given a single static image as input, the model forecasts dense optical flow — the expected pixel-level motion that would occur in the immediate future. The system is trained on tens of thousands of realistic videos without requiring human annotations, leveraging automatically computed optical flow as ground truth. The approach significantly outperforms prior methods across multiple benchmarks and demonstrates the ability to predict plausible motion patterns across diverse scenarios.

Humans possess a remarkable ability to infer future motion from static images: seeing a ball in mid-air, we predict its trajectory; seeing a person mid-stride, we anticipate their next step. This ability relies on learned associations between visual appearance and typical motion patterns. The authors argue that CNNs can learn similar associations from large video datasets, enabling motion prediction from single frames. The task is inherently ambiguous — the same static image can correspond to multiple plausible future motions. Prior approaches have been limited to specific domains (e.g., traffic scenes) or coarse motion categories. This work aims for a general-purpose, pixel-level motion prediction system trained across diverse scenarios.

Optical flow estimation traditionally requires two consecutive frames, with classical methods like Horn-Schunck and modern approaches like DeepFlow. Motion prediction from single images is a less explored direction. Prior work includes action prediction models that classify discrete action categories rather than predicting continuous motion fields, and scene flow estimation methods that require 3D information or stereo input. Random forests have been used for structured pixel-level prediction but are limited in representational capacity compared to deep networks. The proposed CNN approach provides the first general-purpose dense motion prediction from single static images.

The network is a modified seven-layer CNN that takes a 200x200 pixel image as input and produces a 20x20 coarse prediction map. The final layer contains 16,000 neurons (20 x 20 spatial locations x 40 flow clusters), outputting a spatial softmax over flow clusters at each location. The key design choice is to use fully convolutional layers throughout, maintaining spatial correspondence between input and output. The coarse 20x20 output is bilinearly upsampled to the original image resolution for evaluation. Training labels are automatically generated using the DeepFlow algorithm on consecutive video frames, with camera stabilization and averaging over five future frames to reduce label noise.

A crucial methodological choice is to reformulate the optical flow regression problem as a classification task. Rather than directly regressing continuous flow vectors — which tends to produce blurry mean predictions — the authors quantize the optical flow space into 40 clusters using k-means and train the network to predict a probability distribution over these clusters at each pixel using softmax loss. This formulation has two advantages: it avoids averaging to the mean (which destroys directional information when multiple motions are plausible) and preserves directional ambiguity through probability distributions. The expected flow can be recovered by computing the weighted average of cluster centers using the predicted probabilities.

The model is trained on UCF-101 (approximately 350,000 frames) and HMDB-51 (approximately 150,000 frames). Evaluation uses multiple metrics: End-Point-Error (EPE) for magnitude accuracy, direction/orientation similarity for angular accuracy, and Top-N cluster ranking for categorical accuracy. Results demonstrate substantial improvements over Structured Random Forests and Nearest-Neighbor baselines across all three datasets. The method shows strong cross-dataset generalization — a model trained on UCF-101 performs well on HMDB-51 and vice versa, suggesting that the learned motion priors generalize across action types and visual domains. A proof-of-concept multi-frame extension demonstrates that the model can predict motion for multiple future time steps, though with decreasing accuracy.

This paper presents a generalized prediction framework that learns to predict dense optical flow from single static images by training on automatically labeled video data. The key insight that regression as classification preserves motion ambiguity is central to the method's success. The approach enables motion estimation across diverse indoor and outdoor scenarios with multiple agents, demonstrating that static images contain rich predictive information about future dynamics. Future directions include semantic action prediction integration — combining appearance-based motion prediction with action understanding — and pixel-space video synthesis, where predicted flow could drive video generation from single images.