We propose a simple, yet effective approach for spatiotemporal feature learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large scale supervised video dataset. Our findings are threefold: (1) 3D ConvNets are more suitable for spatiotemporal feature learning compared to 2D ConvNets; (2) a homogeneous architecture with small 3x3x3 convolution kernels in all layers is among the best performing architectures; and (3) the learned features, termed C3D, with a simple linear classifier achieve state-of-the-art or near state-of-the-art results on multiple video analysis benchmarks while being compact and efficient to compute.

The need for a generic video descriptor drives this work. An ideal descriptor should possess four properties: genericity across video types, compact representation, computational efficiency, and implementation simplicity. Existing approaches include hand-crafted features (STIPs, HOG3D, improved Dense Trajectories) that are computationally expensive and domain-specific, and 2D CNN features that collapse temporal information. Although 3D ConvNets were proposed before, to our knowledge this work is the first to exploit 3D ConvNets in the context of large-scale supervised training datasets, enabling them to learn truly generic spatiotemporal features.

Prior approaches to video feature learning include hand-crafted spatiotemporal descriptors — Space-Time Interest Points (STIPs), HOG3D, and improved Dense Trajectories (iDT) — which remain competitive but are expensive to compute and require careful engineering. Early deep learning approaches used 2D CNNs on individual frames, discarding temporal information. Two-stream architectures process appearance (RGB) and motion (optical flow) separately, but require pre-computed optical flow and separate networks. The concept of 3D convolution preserves temporal structure but had previously been limited to small datasets and shallow networks.

3D convolution preserves the temporal information of the input signals resulting in an output volume, unlike 2D operations that collapse temporal data. We systematically search for the optimal kernel temporal depth by testing homogeneous depths d = 1, 3, 5, 7. Results show that depth-3 (3x3x3) performs best among the homogeneous architectures. The depth-1 (effectively 2D) variant performs significantly worse, confirming the importance of temporal modeling. The final C3D architecture consists of 8 convolution layers, 5 pooling layers, and 2 fully connected layers, all using 3x3x3 kernels, trained on Sports-1M (1.1 million videos, 487 categories).

C3D features are evaluated across four diverse tasks. On UCF101 action recognition: C3D single net achieves 82.3%, three nets 85.2%, and C3D + iDT reaches 90.4%. C3D outperforms ImageNet and iDT features by 10-20% in low dimensions; at just 10 dimensions, it achieves 52.8% vs. ~32% for baselines. On action similarity (ASLAN): 78.3% accuracy and 86.5% AUC, significantly outperforming the prior best of 68.7%. On scene recognition: Maryland 87.7% (vs. 77.7%), YUPENN 98.1% (vs. 96.2%). C3D runs at 313.9 FPS on GPU — 91x faster than improved Dense Trajectories and 274x faster than Brox's GPU optical flow.

We have demonstrated that 3D ConvNets effectively model appearance and motion simultaneously, establishing C3D as an efficient, compact, and simple-to-use video descriptor applicable across diverse analysis tasks. The homogeneous 3x3x3 architecture trained on Sports-1M produces features that generalize well across action recognition, action similarity, and scene classification. The combination of state-of-the-art accuracy with real-time processing speed makes C3D a practical foundation for video understanding systems. Source code and pre-trained models are publicly available.