The Fisher Vector (FV) representation has become the dominant approach for action recognition in video. In this work, we propose Stacked Fisher Vectors (SFV), which applies the Fisher Vector encoding in multiple layers to progressively capture higher-order statistics of local descriptors. At each layer, Fisher Vectors computed from sub-volumes of the video are aggregated and re-encoded using a new GMM, producing increasingly abstract representations. This hierarchical encoding naturally captures temporal structure and spatial layout at multiple scales. We demonstrate that SFV achieves state-of-the-art results on Hollywood2 and HMDB51 benchmarks, outperforming both flat FV approaches and competing deep learning methods of the time.

Action recognition in video requires capturing both the appearance and motion characteristics of human activities. The standard pipeline involves extracting local spatio-temporal descriptors such as improved dense trajectories (iDT) with HOG, HOF, and MBH features, and then encoding them using Fisher Vector encoding with a Gaussian Mixture Model (GMM). While this approach has been highly successful, the standard FV encoding treats all local descriptors as independently drawn from the same distribution, ignoring the spatial and temporal structure within the video. This flat encoding strategy discards valuable information about how local patterns are organized across space and time.

Spatial pyramid and temporal pyramid pooling strategies have been proposed to incorporate structural information, but they use fixed, predetermined partitioning schemes that may not align with the natural structure of activities. Our Stacked Fisher Vectors approach addresses this by learning a hierarchy of representations through iterative FV encoding, where each layer captures progressively more complex patterns. This is conceptually similar to the hierarchical feature learning in deep neural networks, but operates entirely within the Fisher Vector framework, preserving the strong theoretical properties and computational efficiency of the FV representation.

The Stacked Fisher Vector pipeline proceeds as follows. In the first layer, the video is divided into overlapping spatio-temporal sub-volumes. Within each sub-volume, local descriptors (e.g., iDT features) are encoded using a standard FV with a GMM trained on a large corpus. This produces a set of first-layer FV representations, one per sub-volume. In the second layer, these first-layer FVs are treated as new local descriptors: a second GMM is trained on the collection of first-layer FVs, and each video is re-encoded using this new GMM to produce a second-layer FV. This process can be repeated for additional layers. The final representation is the concatenation of FVs from all layers, capturing information at multiple levels of abstraction.

Several design choices are crucial for the success of SFV. First, the sub-volume partitioning strategy determines the spatial and temporal granularity: we use a multi-scale approach with overlapping sub-volumes at different temporal and spatial resolutions. Second, dimensionality reduction via PCA is applied to the first-layer FVs before second-layer encoding, reducing the input dimension while preserving the most informative directions. Third, the FVs at each layer are power-normalized and L2-normalized before being used as input to the next layer or the final classifier, following established best practices for FV representations.

We evaluate SFV on Hollywood2 (12 action classes) and HMDB51 (51 action classes). On Hollywood2, SFV achieves mAP of 66.8%, outperforming standard FV (64.3%) and spatial pyramid FV (65.1%). On HMDB51, SFV reaches accuracy of 57.2%, surpassing flat FV (55.9%) and temporal pyramid FV (56.0%). Compared to early deep learning approaches for action recognition, SFV outperforms two-stream CNNs (56.8% on HMDB51) of the time. The consistent improvements across both datasets confirm that hierarchical encoding captures complementary information beyond what flat encoding provides.

Layer-wise analysis reveals that the second layer provides the most significant improvement (1.5-2.0% over single-layer FV), while the third layer adds only marginal gains (0.2-0.5%). This suggests that two layers of stacking capture most of the structural information, and further stacking yields diminishing returns. We also analyze the effect of sub-volume granularity: finer partitioning captures more local structure but increases computational cost. The optimal trade-off is achieved with 4-8 temporal segments and 2x2 spatial grids, which provides sufficient structural information without excessive fragmentation.

We have introduced Stacked Fisher Vectors, a hierarchical encoding approach that extends the standard Fisher Vector framework by iteratively encoding sub-volume FVs to capture multi-level temporal and spatial structure. SFV achieves state-of-the-art results on major action recognition benchmarks while remaining within the well-understood FV framework. Our work demonstrates that hierarchical representation learning is beneficial even within traditional feature encoding pipelines, and provides a complementary perspective to end-to-end deep learning approaches for video understanding.