Deep Recurrent Neural Network architectures, though remarkably capable at modeling sequences, lack an intuitive high-level spatio-temporal structure. We propose a method that combines the power of spatio-temporal graphs with the sequence modeling ability of RNNs, resulting in a model that can better capture the structural properties of real-world spatio-temporal problems. Our approach transforms arbitrary spatio-temporal graphs into a rich RNN mixture that is feedforward, fully differentiable, and jointly trainable. The method is generic and principled, applicable across diverse tasks. We demonstrate improvements over existing approaches in human activity recognition, human motion modeling, and driver maneuver anticipation.

Many real-world problems involve spatio-temporal data with rich structural dependencies. For example, in human activity recognition, the human body can be represented as a graph of joints, where the spatial relationships between joints and their temporal evolution are crucial for understanding the activity. Recurrent Neural Networks (RNNs), particularly LSTMs, have become the standard approach for sequence modeling tasks. However, standard RNNs treat the input as a flat vector at each time step, ignoring the rich spatial structure of the data. This forces the network to implicitly learn the structural relationships from data, which is both inefficient and may not generalize well.

Prior attempts to incorporate structure into RNNs have been task-specific and ad hoc, designing custom architectures for each new problem. In contrast, we propose Structural-RNN (S-RNN), a generic framework that can transform any spatio-temporal graph into a feedforward mixture of RNNs. The key insight is that the nodes and edges of the spatio-temporal graph can be mapped to specific RNN components — nodeRNNs model the temporal evolution of individual entities, while edgeRNNs model the interactions between entities. The resulting architecture is fully differentiable and can be trained end-to-end using backpropagation through time.

Graphical models such as Conditional Random Fields (CRFs) and Markov Random Fields (MRFs) have long been used to model structured dependencies in computer vision and robotics. However, they typically rely on hand-crafted features and limited expressive power of the potential functions. On the other hand, deep learning methods, especially RNNs, have shown remarkable success in sequence modeling but do not explicitly leverage structural priors. Recent works have attempted to combine graph structures with neural networks through graph neural networks, but most are limited to static graphs or specific domains. Our work bridges this gap by providing a principled mapping from spatio-temporal graphs to RNN architectures.

A spatio-temporal graph (st-graph) is defined as a graph G = (V, E) where nodes V represent semantic entities (e.g., human body parts, vehicles, objects in a scene) and edges E represent spatial or temporal relationships between them. The graph evolves over time: at each time step, spatial edges capture the interactions between co-existing entities, while temporal edges link the same entity across consecutive time steps. This representation provides a rich, structured prior that encodes domain knowledge about the problem, going beyond the flat vector assumption of standard RNNs.

The transformation from a spatio-temporal graph to a Structural-RNN proceeds in two steps. First, we factor the st-graph into a set of node-level and edge-level components. Each node type gets a nodeRNN that models the temporal evolution of that entity, and each edge type gets an edgeRNN that models the pairwise interaction between connected entities. Second, we wire these RNNs together according to the graph structure: at each time step, edgeRNNs receive the hidden states of their connected nodeRNNs, and nodeRNNs aggregate information from their adjacent edgeRNNs. This creates a feedforward computational graph that can be unrolled over time and trained with standard backpropagation through time (BPTT). Crucially, entities of the same semantic type share weights, reducing the parameter count and enabling generalization.

We evaluate Structural-RNN on three diverse tasks. For human activity recognition on the CAD-120 dataset, S-RNN achieves state-of-the-art performance with 89.2% accuracy, outperforming both graphical model baselines (CRF-based methods) and standard LSTM approaches. For human motion forecasting on the H3.6M dataset, S-RNN generates more realistic motion predictions than LSTM-3LR and ERD baselines, as validated by both quantitative metrics and user studies where "S-RNN generates most realistic human motions majority of the times." For driver maneuver anticipation, the model successfully anticipates turns and lane changes several seconds before they occur. Training employs backpropagation through 100 time steps with mini-batch size of 100 sequences and gradient clipping at L2-norm 25.0.

We have presented Structural-RNN, a generic and principled approach for combining the representational power of spatio-temporal graphs with the learning capacity of recurrent neural networks. By providing a systematic mapping from st-graphs to mixtures of nodeRNNs and edgeRNNs, our framework enables researchers and practitioners to incorporate structural domain knowledge into deep sequence models without designing custom architectures. Experiments across activity recognition, motion forecasting, and maneuver anticipation demonstrate the versatility and effectiveness of the approach. The resulting models are feedforward, fully differentiable, and jointly trainable, making them readily applicable to new spatio-temporal problems.