This paper introduces NTU RGB+D, a large-scale dataset for 3D human activity analysis containing 56,880 video samples and over 4 million frames, collected from 40 distinct subjects. The dataset includes 60 action classes covering daily activities, mutual actions, and health-related actions, captured using Microsoft Kinect v2 sensors. Four data modalities are provided: RGB video (1920x1080), depth maps (512x424), 3D skeletal data (25 body joints), and infrared sequences. The authors also propose a Part-aware LSTM (P-LSTM) model that leverages body part dynamics for improved action classification, demonstrating advantages of deep learning methods over hand-crafted features.

Human activity recognition is a fundamental task in computer vision with applications in surveillance, human-computer interaction, and healthcare. While significant progress has been made using RGB video data, existing datasets are limited in scale, number of action categories, and diversity of viewpoints. The recent availability of depth sensors like Microsoft Kinect has opened new possibilities by providing 3D structural information that is invariant to illumination changes. However, current RGB+D datasets such as MSR-DailyActivity3D and UTKinect contain fewer than 1,000 samples, severely limiting the application of data-hungry deep learning methods.

Existing RGB+D action recognition datasets include MSR-DailyActivity3D (320 samples, 16 classes), Northwestern-UCLA (1,475 samples, 10 classes), and UWA3D Multiview Activity II (1,075 samples, 30 classes). These datasets suffer from limited subject diversity, small number of viewpoints, and insufficient scale for training deep models. On the methodology side, approaches can be categorized into hand-crafted features (e.g., HON4D, Actionlet Ensemble) and deep learning methods (e.g., Recurrent Neural Networks on skeleton sequences). Most deep learning approaches for skeleton-based recognition use vanilla LSTMs that treat all joints equally, ignoring the compositional structure of the human body.

The NTU RGB+D dataset captures 56,880 video samples across 60 action classes from 40 human subjects (ages 10-35) using three Microsoft Kinect v2 cameras simultaneously. The cameras capture three different horizontal views at angles of -45 degrees, 0 degrees, and +45 degrees, resulting in 80 distinct camera viewpoints across 17 different setups varying camera heights (0.5-2.7m) and distances (2.0-4.5m). Four modalities are provided: RGB frames (1920x1080), depth maps (512x424), 3D skeletal joint positions (25 joints per body, up to 2 bodies), and infrared video.

Two standard evaluation protocols are defined. The Cross-Subject protocol splits 40,320 training samples from 20 subjects and 16,560 test samples from the remaining 20 subjects. The Cross-View protocol uses 37,920 samples from cameras 2 and 3 for training and 18,960 samples from camera 1 for testing. These protocols evaluate a method's ability to generalize across different people and different viewpoints, respectively, which are critical for real-world deployment.

The proposed Part-aware LSTM (P-LSTM) divides the memory cell into part-based sub-cells for five body regions: torso, left arm, right arm, left leg, and right leg. Each body part maintains independent input and forget gates while sharing an output gate, enabling the network to "learn the long-term patterns specifically for each body part". This design reflects the observation that different body parts contribute differently to different actions — for example, arm movements dominate in "waving" while leg dynamics are critical for "kicking."

Extensive experiments are conducted on the NTU RGB+D dataset. The 2-Layer P-LSTM achieves 62.93% accuracy on the Cross-Subject protocol and 70.27% on the Cross-View protocol, outperforming both standard LSTM (60.69% and 67.29%) and hand-crafted methods such as FTP Dynamic Skeletons (60.23% and 65.22%). The results demonstrate that part-aware modeling provides consistent improvements over vanilla LSTM. Notably, the dataset remains challenging even for the best methods, with Cross-Subject accuracy below 63%, indicating significant room for improvement.

This paper presents NTU RGB+D, the largest RGB+D dataset for human activity recognition, providing 56,880 samples with four complementary modalities and two standardized evaluation protocols. The proposed Part-aware LSTM demonstrates that incorporating body part structure into recurrent models improves skeleton-based action recognition. The dataset's scale and multi-modal nature make it a valuable benchmark for developing and evaluating next-generation action recognition methods, particularly data-driven deep learning approaches.