We propose a novel skeletal representation that explicitly models the 3D geometric relationships between various body parts using rotations and translations in 3D space. Since the relative 3D geometry between a pair of body parts can be described by an element of the special Euclidean group SE(3), the skeletal representation of a human body is a point in the Lie group SE(3) x ... x SE(3). An action is then represented as a curve in this Lie group. We map these curves to the corresponding Lie algebra, which is a vector space, and perform classification using a combination of dynamic time warping, Fourier temporal pyramid representation, and linear SVM. Experiments demonstrate superior performance compared to existing skeleton-based approaches on multiple action recognition datasets.

Human action recognition from 3D skeletal data has gained significant attention due to the availability of cost-effective depth sensors such as Microsoft Kinect. These sensors provide real-time estimation of 3D joint positions, offering a compact yet informative representation of human body configuration. However, most existing skeleton-based methods treat joint positions as points in Euclidean space, ignoring the underlying geometric structure of the human body — specifically, the fact that body parts are rigid segments connected by joints that allow rotation and translation.

The key insight of our approach is that the relative 3D rigid body transformation between any pair of body parts can be naturally represented as an element of SE(3), the special Euclidean group consisting of 3D rotations (SO(3)) and translations. By considering all relevant body part pairs, the entire skeleton configuration at each time frame becomes a point in a product Lie group. An action sequence is therefore a curve in this Lie group manifold. Since Lie groups are non-Euclidean spaces where standard machine learning tools cannot be directly applied, we leverage the exponential and logarithmic maps to transfer curves to the tangent space — the Lie algebra — where Euclidean methods become applicable.

Skeleton-based action recognition methods can be broadly categorized into joint-based and body-part-based approaches. Joint-based methods represent actions using raw joint coordinates or their temporal derivatives, which are sensitive to body size variations and viewpoint changes. Body-part-based methods use relative joint positions or angular representations, offering better invariance but lacking a principled mathematical framework for modeling the geometric structure. Recent works have explored covariance descriptors on Riemannian manifolds for action recognition, but these do not exploit the specific Lie group structure inherent in rigid body transformations.

Given a skeleton with n body parts, we represent the relative geometry between body part i and body part j as a rigid body transformation (R_ij, t_ij) in SE(3), where R_ij is a 3x3 rotation matrix in SO(3) and t_ij is a 3D translation vector. This transformation maps the local coordinate frame of part i to that of part j. By considering a set of C body part pairs, the configuration at each time frame t is represented as a point p(t) = (g_1(t), g_2(t), ..., g_C(t)) in the product Lie group G = SE(3)^C. The full action is a curve {p(t), t=1,...,T} in G.

Since the Lie group G is a curved manifold, not a vector space, standard linear classifiers cannot be applied directly. We use the logarithmic map at the identity element to project each point p(t) from G to the Lie algebra g, which is the tangent space at the identity — a flat vector space. For each SE(3) component, the logarithmic map produces a 6-dimensional vector (3 for rotation via the matrix logarithm of SO(3), and 3 for translation). The entire skeleton at time t is thus mapped to a 6C-dimensional vector in the Lie algebra. This mapping preserves the essential geometric information while enabling the use of Euclidean machine learning tools.

After mapping each time frame to the Lie algebra, an action sequence becomes a time series of 6C-dimensional vectors. To handle temporal misalignment and varying action speeds, we employ dynamic time warping (DTW) to compute distances between sequences. For classification, we construct a Fourier temporal pyramid (FTP) representation that captures both global and local temporal patterns at multiple resolutions. The FTP features are then fed to a linear SVM for final classification. This combination allows the system to handle actions of different durations while capturing the essential temporal dynamics.

We evaluate on three standard benchmarks: MSR Action3D, Florence 3D Actions, and UTKinect-Action datasets. On MSR Action3D, our method achieves 92.46% accuracy, outperforming the previous best skeleton-based method by a significant margin. On Florence 3D Actions, we obtain 90.88% accuracy, and on UTKinect-Action, 97.08% accuracy. Ablation studies show that the Lie group representation consistently outperforms the raw joint coordinate representation across all datasets, confirming that modeling the geometric structure of the skeleton is beneficial. The choice of body part pairs also significantly impacts performance, with a combination of adjacent and non-adjacent pairs yielding the best results.

We have proposed a principled approach to skeleton-based action recognition that represents 3D skeletons as points in the Lie group SE(3)^C. By mapping action curves from the Lie group to the Lie algebra via the logarithmic map, we enable the use of standard Euclidean classifiers while preserving the geometric structure of the human body. Experimental results on multiple benchmarks demonstrate the effectiveness of this representation. Future directions include learning the optimal body part pairs from data and extending the framework to incorporate appearance features alongside skeletal geometry.