We present a novel descriptor for activity recognition from depth sequences. Previous approaches typically treat shape and motion independently, computing features on spatial and temporal dimensions separately. In contrast, we describe the depth sequence as a surface in 4D space (x, y, depth, time) and capture the distribution of surface normal orientations in this 4D volume jointly. We propose the Histogram of Oriented 4D Normals (HON4D) descriptor, which quantizes the 4D normals using 4D projectors initialized from a regular polychoron and refined using discriminative density measures. Our approach outperforms the state-of-the-art on all relevant benchmarks, demonstrating the advantage of joint shape-motion representation.

Activity recognition is a fundamental problem in computer vision with applications in surveillance, human-computer interaction, and assistive robotics. The advent of affordable depth sensors has shifted the field from purely appearance-based methods to approaches that can leverage 3D structural information. However, most existing depth-based methods process shape (spatial structure) and motion (temporal dynamics) as separate components, typically extracting spatial features from individual frames and temporal features from frame differences or optical flow.

We argue that shape and motion are intrinsically coupled in human activities and should be captured jointly rather than independently. To this end, we propose treating the depth video as a surface embedded in 4D space, where the four dimensions correspond to spatial coordinates (x, y), depth (z), and time (t). The orientation of the surface normal at each point in this 4D volume encodes both the local shape and the local motion simultaneously. A horizontal normal indicates a spatial boundary, a normal pointing in the time direction indicates temporal change, and oblique normals capture coordinated shape-motion patterns.

Depth-based activity recognition has grown rapidly since the introduction of the Kinect sensor. Skeleton-based methods such as those using joint angle features or skeletal quads have shown strong results but rely on accurate skeleton estimation, which can fail under occlusion or unusual poses. Depth map-based approaches include bag-of-3D-points, random occupancy patterns, and space-time occupancy patterns. These methods typically quantize spatial and temporal information separately, missing the joint structure. HOG3D extends histograms of oriented gradients to 3D (x, y, t) but does not incorporate the depth dimension as a geometric coordinate.

Given a depth video sequence, we represent it as a 4D volume V(x, y, z, t) where each point (x, y) in frame t has an associated depth value z. We treat this as a hypersurface in 4D Euclidean space. At each occupied point, we compute the 4D surface normal by taking the cross product of three tangent vectors along the x, y, and t directions. The resulting normal vector n = (n_x, n_y, n_z, n_t) lives on the unit 3-sphere S^3 in 4D. The key insight is that this single 4D normal simultaneously encodes spatial geometry (via n_x, n_y, n_z) and temporal dynamics (via n_t), providing a unified shape-motion representation.

The HON4D descriptor is constructed by quantizing the 4D normal directions and accumulating a histogram over a spatiotemporal volume. Unlike the 3-sphere S^3 in 3D where orientations can be evenly sampled using an icosahedron, uniform sampling on S^3 in 4D requires a regular polychoron (4D polytope). We use the 600-cell, which has 120 vertices uniformly distributed on S^3, as the initial set of 4D projectors. Each 4D normal is assigned to its nearest projector, building a 120-bin histogram that captures the distribution of oriented 4D normals within the volume.

While the uniform polychoron-based projectors provide a good initialization, they are not optimized for the specific activity recognition task. We propose to refine the projector locations on S^3 using a discriminative criterion. Specifically, we define a density-based discriminability measure that evaluates how well a set of projectors separates the 4D normal distributions of different activity classes. We use a gradient-based optimization procedure to iteratively adjust projector positions, moving them toward regions of S^3 where different activities exhibit maximally distinct normal distributions. This data-driven refinement improves classification accuracy by 3-5% across benchmarks.

We evaluate HON4D on three benchmark datasets: the MSR Action3D dataset (20 actions, 10 subjects), the MSR Gesture3D dataset (12 hand gestures), and the MSR DailyActivity3D dataset (16 daily activities). On MSR Action3D, HON4D achieves 88.89% accuracy compared to 86.50% for the previous best method (Actionlet Ensemble). On MSR Gesture3D, we achieve 92.45% compared to 89.20% for previous state-of-the-art. On the challenging MSR DailyActivity3D, we achieve 80.00% versus 68.00% for the next best approach, a 12 percentage point improvement. Ablation studies show that removing the discriminative projector refinement reduces accuracy by 3-5%, and that the 4D representation outperforms 3D (x,y,t) normals by 4-8%.

We have presented HON4D, a novel descriptor that captures joint shape-motion information by computing histograms of oriented surface normals in 4D space. By treating depth sequences as surfaces in a 4D volume and using discriminatively refined polychoron-based projectors, our approach achieves state-of-the-art results on multiple activity recognition benchmarks. The results confirm that jointly modeling shape and motion through 4D normals is more effective than treating them independently. Future work includes extending to more complex activities with multiple interacting subjects and investigating learned representations in the 4D normal space.