In this paper we discuss several forms of spatiotemporal convolutions for video analysis and study their effects on action recognition. Our motivation stems from the observation that 3D CNNs with 3D convolution kernels outperform 2D CNNs when used in residual learning frameworks. We show that factorizing the 3D convolutional filters into separate spatial and temporal components yields significantly better accuracy. Specifically, our proposed R(2+1)D block decomposes a 3D convolution into a 2D spatial convolution followed by a 1D temporal convolution. This decomposition doubles the number of nonlinearities in the network, which we argue is a key factor behind the improved performance. We achieve state-of-the-art results on Sports-1M, Kinetics, UCF101, and HMDB51.

Video understanding requires models that can capture both spatial appearance and temporal dynamics. Early deep learning approaches for video applied 2D CNNs to individual frames and aggregated temporal information through pooling or recurrent networks, which limits the ability to model fine-grained temporal patterns. 3D convolutions (C3D) directly process spatiotemporal volumes but are computationally expensive and harder to train. Recent work has shown mixed results: some studies find 3D convolutions offer no advantage over 2D, while others demonstrate clear benefits. We aim to systematically study the design space of spatiotemporal convolutions to resolve these contradictions.

Two-stream networks process RGB frames and optical flow separately, achieving strong results but requiring pre-computed optical flow. C3D proposed using 3D convolutions with 3x3x3 kernels for spatiotemporal feature learning, but was limited by the VGG-style architecture without residual connections. I3D inflated successful 2D architectures (Inception) to 3D and achieved strong results on Kinetics, but the question of how to optimally structure spatiotemporal convolutions within modern residual architectures remains open. P3D and S3D explored various decomposition strategies but did not provide a comprehensive comparative analysis.

We consider five forms of spatiotemporal convolutions within a ResNet-based architecture. R2D: uses only 2D convolutions (1x3x3 kernels), treating each frame independently. R3D: uses full 3D convolutions (3x3x3 kernels) throughout. MC_x: uses 3D convolutions in early layers and 2D in later layers (mixed convolutions). rMC_x: the reverse — 2D in early layers, 3D in later layers. R(2+1)D: decomposes every 3D convolution into a 2D spatial convolution followed by a 1D temporal convolution. All five architectures have approximately the same number of parameters, enabling fair comparison.

The R(2+1)D block replaces a full 3D convolution of size t x d x d with a spatial 2D convolution of size 1 x d x d followed by a temporal 1D convolution of size t x 1 x 1. Between the two operations, a ReLU nonlinearity and batch normalization are inserted. This decomposition has two advantages. First, it doubles the number of nonlinear activations in the network, which increases the complexity of functions that can be represented. Second, it facilitates the optimization: learning spatial and temporal features separately is easier than learning them jointly, as the error surface becomes smoother with the decomposition. The intermediate feature dimension M_i is chosen to match the parameter count of the equivalent full 3D convolution.

We evaluate all five architectures on Sports-1M, Kinetics, UCF101, and HMDB51 using clip-level and video-level accuracy. On Kinetics with 18-layer ResNet, R(2+1)D achieves clip-level accuracy of 72.0%, outperforming R3D (71.1%), MC3 (69.6%), rMC3 (69.0%), and R2D (64.8%). The advantage of R(2+1)D over R3D is consistent across different depths: with a 34-layer ResNet, R(2+1)D reaches 74.3% vs. R3D at 72.7%. When pre-trained on Sports-1M and fine-tuned, R(2+1)D achieves 97.3% on UCF101 and 78.7% on HMDB51, competitive with the state-of-the-art I3D results of 98.0% and 80.7% (which uses additional optical flow input).

We provide training curve analysis comparing R3D and R(2+1)D. The R(2+1)D model consistently shows lower training loss throughout the training process, suggesting that the decomposition indeed facilitates optimization. Furthermore, the gap between training and validation loss is smaller for R(2+1)D, indicating better generalization rather than merely better fitting to the training data. This supports our hypothesis that the factorized architecture leads to a smoother optimization landscape.

We have presented a systematic study of spatiotemporal convolutions for video action recognition. Our key finding is that R(2+1)D — decomposing 3D convolutions into separate spatial and temporal components — consistently outperforms full 3D convolutions across multiple datasets and network depths. We attribute this to the increased number of nonlinearities and the simplified optimization landscape. The R(2+1)D decomposition is a simple drop-in replacement that can benefit any architecture using 3D convolutions, making it a practical and widely applicable contribution to video understanding.