Human action recognition from skeleton data has attracted increasing attention due to the availability of depth sensors and real-time pose estimation algorithms. Existing methods typically concatenate all joint coordinates into a single feature vector, ignoring the spatial structure of the human skeleton. In this paper, we propose a hierarchical recurrent neural network (HRNN) that decomposes the human skeleton into five anatomical groups (two arms, two legs, and a torso) and processes them in a bottom-up hierarchy. At the lowest level, individual body part subnets capture part-specific motion patterns. At higher levels, these part representations are progressively fused to form a full-body representation. This hierarchical decomposition naturally encodes the physical structure of the human body into the network architecture. We demonstrate state-of-the-art performance on several benchmark datasets.

Skeleton-based action recognition has emerged as an important research direction due to several advantages: skeleton data is compact, view-invariant to a large extent, and robust to background clutter and illumination changes. With the popularization of Microsoft Kinect and advances in real-time pose estimation, large-scale skeleton datasets have become available. However, most existing approaches treat the skeleton as a flat vector of joint coordinates, applying standard classifiers (SVMs, random forests) or feeding the entire vector into a single RNN. This ignores the rich structural information inherent in the human body's kinematic tree.

We propose to exploit the physical structure of the human body by designing a hierarchical RNN architecture. The key observation is that human actions are composed of coordinated movements of body parts: for instance, "walking" involves coordinated leg motions, "clapping" involves coordinated arm motions, and "bowing" primarily involves torso motion. By first extracting part-level temporal representations and then combining them hierarchically, the network can learn both part-specific patterns and inter-part coordination at different levels of abstraction.

Hand-crafted feature approaches for skeleton action recognition include covariance descriptors, Lie group representations, and actionlets. While effective, these methods require domain expertise to design features and may not generalize across datasets. Deep learning approaches have begun to be applied: Du et al. used standard RNNs on concatenated joint vectors, and Veeriah et al. explored differential RNNs. However, these methods do not explicitly model the spatial structure of the skeleton. Graph-based approaches model joint relationships but typically use static graph structures that do not capture temporal dynamics. Our HRNN uniquely combines structural decomposition with recurrent temporal modeling in an end-to-end trainable framework.

The human skeleton (typically 20-25 joints from a Kinect sensor) is decomposed into five anatomical groups: left arm (shoulder, elbow, wrist, hand), right arm, left leg (hip, knee, ankle, foot), right leg, and torso (spine, neck, head). Each group contains the 3D coordinates of its constituent joints at each time step. This decomposition follows the natural kinematic chain of the human body — joints within the same group tend to move in a coordinated manner, while inter-group coordination encodes higher-level action semantics.

The hierarchical RNN consists of multiple layers. At the first layer, five independent bidirectional RNNs (one per body part group) process the temporal sequences of their respective joint coordinates. Each part-level RNN produces a hidden state sequence capturing the temporal dynamics of that body part. At the second layer, the hidden states from related parts are concatenated and fed into higher-level RNNs: the left arm and right arm representations are combined, the left leg and right leg are combined, and the torso representation is carried forward. At the third (top) layer, all representations are fused into a single RNN that produces the final full-body temporal representation. The last hidden state of the top-level RNN is fed into a softmax classifier for action classification.

Each RNN in the hierarchy uses bidirectional connections to capture both forward and backward temporal context. We employ Long Short-Term Memory (LSTM) units to address the vanishing gradient problem inherent in standard RNNs when processing long sequences. The LSTM's gating mechanism (input gate, forget gate, output gate) allows the network to selectively remember or forget information across time steps. For action recognition, this is particularly important because discriminative motion patterns may occur at any point in the sequence, and the network must learn to attend to the most informative temporal segments.

We evaluate on three benchmark datasets: NTU RGB+D (the largest skeleton dataset at the time with 56,880 sequences and 60 action classes), SBU Kinect Interaction, and CMU Motion Capture. On NTU RGB+D, our HRNN achieves significant improvements over flat RNN baselines in both cross-subject and cross-view evaluation protocols. Compared to hand-crafted feature methods (Lie Group, Skeletal Quads, Dynamic Skeletons), HRNN outperforms all baselines by clear margins. On SBU Kinect Interaction (two-person interactions), the hierarchical architecture also demonstrates superior performance, showing that the part-level decomposition is beneficial even for interaction recognition.

Ablation studies validate the hierarchical design: the three-layer HRNN consistently outperforms a single-layer flat RNN that receives all joints simultaneously. The part-level decomposition alone (first layer only) already provides improvements over the flat baseline, confirming that respecting the body's spatial structure is beneficial. The additional fusion layers further improve performance by capturing inter-part coordination. Analysis of per-action performance reveals that the hierarchical model is especially beneficial for actions involving specific body parts (e.g., "waving" benefits from arm-specific processing), while maintaining competitiveness on full-body actions.

We have presented a hierarchical recurrent neural network for skeleton-based action recognition that explicitly encodes the physical structure of the human body into the network architecture. By decomposing the skeleton into anatomical groups and processing them in a bottom-up hierarchy, the network learns part-specific temporal patterns and inter-part coordination at multiple levels of abstraction. Experimental results on multiple benchmarks demonstrate the effectiveness of this structure-aware approach. The proposed framework is general and could be extended to other articulated structures beyond the human body, such as animal locomotion or robotic manipulation.