Deep convolutional networks have achieved great success for visual recognition in still images. However, for action recognition in videos, the advantage over traditional methods is not so evident. This paper aims to discover the principles to design effective ConvNet architectures for action recognition in videos and proposes temporal segment networks (TSN), a novel framework for video-level action recognition based on the idea of long-range temporal structure modeling. Our approach combines a sparse temporal sampling strategy and video-level supervision to enable efficient and effective learning using the whole action video. We achieve the state-of-the-art performance on the HMDB51 (69.4%) and UCF101 (94.2%) action recognition benchmarks, demonstrating the effectiveness of temporal segment networks and the proposed good practices.

Action recognition is a challenging problem due to the large variations in appearance, viewpoint, and temporal structure of actions. The two-stream architecture proposed by Simonyan and Zisserman processes RGB frames and optical flow separately through two ConvNets. However, this architecture only considers short-term temporal information within a single frame or a short stack of optical flow. It does not model the long-range temporal structure that is critical for distinguishing many action categories. We propose to address this limitation through a temporal segment network that divides the video into segments and aggregates information across segments, effectively capturing the temporal structure of the entire video.

The key idea of TSN is to divide a video into K equal segments and randomly sample one snippet from each segment. Each snippet is processed by a shared-weight ConvNet to produce a class prediction. These K predictions are then aggregated using a consensus function (e.g., averaging) to produce a video-level prediction. Formally, given a video V divided into K segments {S_1, S_2, ..., S_K}, we sample a snippet T_k from each segment. The class score is: G(T_1, T_2, ..., T_K) = g(F(T_1; W), F(T_2; W), ..., F(T_K; W)), where F is the ConvNet function, W the shared parameters, and g the segment consensus function. This design enables efficient modeling of the whole video using only a sparse set of frames.

We also study several good practices for training deep ConvNets for action recognition. These include: (1) cross-modality pre-training — initializing the optical flow ConvNet with RGB pre-trained weights by modifying the first convolution layer; (2) regularization techniques — using batch normalization with dropout and partial BN (freezing mean and variance of all BN layers except the first one); (3) data augmentation — corner cropping and multi-scale cropping to increase data diversity. These practices collectively address the over-fitting problem that is common when training deep networks on relatively small action recognition datasets.

We evaluate TSN on UCF101 and HMDB51 benchmarks. With BN-Inception as the base architecture, TSN achieves 94.2% on UCF101 and 69.4% on HMDB51, setting new state-of-the-art results. Ablation studies confirm: (1) increasing K from 1 to 7 segments consistently improves performance; (2) cross-modality pre-training improves the flow stream by 3.5%; (3) the combination of RGB, optical flow, and warped flow modalities yields the best results. Compared to the original two-stream approach, TSN improves by 6.3% on UCF101 and 10.1% on HMDB51.

We have presented temporal segment networks, a video-level framework for action recognition that effectively models long-range temporal structure through sparse sampling and segment consensus. Together with a set of good practices for training, TSN achieves state-of-the-art results on major action recognition benchmarks. Our framework is general and can be applied with different base architectures and input modalities.