Dense trajectories have been shown to be an efficient video representation for action recognition. In this paper, we improve the dense trajectory approach by explicitly estimating camera motion and removing it from the optical flow. We use human detection and feature point matching to robustly estimate homographies between consecutive frames. The estimated camera motion is used to cancel out background motion in the optical flow and trajectory shape descriptors. We also propose to remove trajectories consistent with camera motion rather than including them as features. Our improved trajectories yield significant improvements over the original dense trajectories on Hollywood2 (64.3%), HMDB51 (57.2%), and UCF50 (91.2%), achieving state-of-the-art results across these benchmarks.

Action recognition in video has received considerable attention due to its applications in video surveillance, human-computer interaction, and video retrieval. The dense trajectory framework proposed by Wang et al. (2011) achieves strong performance by densely sampling feature points and tracking them using optical flow, then computing local descriptors along the trajectories. However, a key limitation is that dense trajectories do not distinguish between camera-induced motion and object motion. When the camera moves (pan, tilt, zoom), all trajectories are affected, and background trajectories carry no information about the action but dominate the representation. This motivates our approach: by explicitly estimating and compensating for camera motion, we can extract trajectories that truly reflect human actions.

Local spatio-temporal features such as STIP (Space-Time Interest Points) and cuboid features have been widely used for action recognition. The bag-of-visual-words pipeline with HOG, HOF, and MBH descriptors remains the dominant paradigm. Dense trajectories outperform interest-point-based approaches by providing denser coverage and motion-aware sampling. Attempts to handle camera motion include motion stabilization and background subtraction, but these are either too aggressive (removing useful motion) or require static cameras. Our approach is more principled: we estimate a parametric camera motion model and use it to selectively compensate trajectories and descriptors.

The dense trajectory framework densely samples feature points on a regular grid at multiple spatial scales. Points are tracked for L = 15 frames using median filtering of optical flow. Along each trajectory, four types of descriptors are computed within a spatio-temporal volume: trajectory shape (displacement vectors), HOG (Histogram of Oriented Gradients), HOF (Histogram of Optical Flow), and MBH (Motion Boundary Histograms). Descriptors are encoded using Fisher vectors with GMM vocabularies of 256 Gaussians, and classification is performed with linear SVMs.

We estimate camera motion between consecutive frames by computing a homography using RANSAC on matched feature points. To make the estimation robust to foreground motion (humans performing actions), we employ two strategies: (1) we use a human detector to identify and exclude foreground regions from the feature matching, and (2) we rely on RANSAC's outlier rejection to handle remaining foreground points. The estimated homography H captures the dominant planar camera motion (pan, tilt, rotation). We then warp each frame by the inverse homography to obtain a camera-motion-compensated optical flow. This compensated flow ideally captures only the independent motion of objects in the scene.

With the estimated camera motion, we make three improvements to the descriptors: (1) Trajectory shape descriptors are computed from the compensated optical flow, so they encode only object motion; (2) HOF descriptors are computed from the compensated flow rather than the raw flow; (3) Trajectories that are consistent with the estimated camera motion (background trajectories) are removed entirely. The MBH descriptor, which already computes derivatives of optical flow, is relatively insensitive to global translational camera motion and thus benefits less from compensation. However, even MBH shows improvement when camera rotation is significant.

We evaluate on three challenging benchmarks: Hollywood2 (12 action classes from movies), HMDB51 (51 action classes from diverse sources), and UCF50 (50 action classes from YouTube). Our improved trajectories achieve 64.3% on Hollywood2 (vs. 58.2% for original dense trajectories), 57.2% on HMDB51 (vs. 48.3%), and 91.2% on UCF50 (vs. 85.6%). The improvements are largest on datasets with significant camera motion (Hollywood2 movies involve extensive camera movement). Ablation studies confirm that (1) camera motion compensation in trajectory shape and HOF descriptors provides the most benefit, (2) removing background trajectories further improves performance, and (3) combining all improved descriptors with Fisher vectors achieves the best results. Our method achieves state-of-the-art on all three benchmarks at the time of publication.

We have presented improved trajectories for action recognition by explicitly estimating and compensating for camera motion. Our approach consistently improves upon the original dense trajectory framework across multiple challenging benchmarks, with the largest gains on videos with significant camera motion. The key insight is that separating camera-induced motion from true object motion leads to more discriminative representations. The approach is simple, effective, and compatible with the standard bag-of-words pipeline. Future directions include incorporating depth information for more accurate camera estimation and exploring deep learning approaches for learning motion-compensated representations end-to-end.