In this work, we establish dense correspondences between an RGB image and a surface-based representation of the human body. We first gather dense correspondences for 50K persons appearing in the COCO dataset by introducing an efficient annotation pipeline. We then use our dataset to train CNN-based systems that deliver dense correspondence "in the wild," namely in the presence of background clutter, occlusions, and scale variations. We experiment with region-based and fully-convolutional architectures and observe that region-based models substantially outperform fully-convolutional approaches. We further use a "teacher" inpainting network to address the challenge of sparse supervision, resulting in improvements. Our system achieves real-time performance at 20-26 frames per second.

Understanding humans in images is a fundamental problem in computer vision. Existing approaches typically focus on sparse representations such as 2D keypoints or 3D skeletal poses. While these provide useful information, they fail to capture the full complexity of human body surface geometry. We argue that a dense, surface-based correspondence map provides a substantially richer description of human pose and body shape, enabling applications in augmented reality, graphics, and human-computer interaction.

The key challenge is the lack of large-scale datasets with dense surface correspondence annotations. Collecting such annotations is substantially more complex than labeling keypoints. We address this through a carefully designed two-stage annotation pipeline: first, annotators perform semantic body part segmentation; second, they identify surface correspondences within each part using pre-rendered 3D body views. This pipeline allows us to efficiently annotate the DensePose-COCO dataset with over 5 million correspondence points across 50K persons.

Human pose estimation has progressed from pictorial structure models to deep learning approaches. Convolutional Pose Machines and Stacked Hourglass Networks have achieved impressive results for 2D keypoint detection. Body model fitting approaches like SMPL reconstruct 3D body meshes from images, but typically operate in a top-down fashion requiring person detection and are computationally expensive. Dense correspondence estimation has been explored between image pairs, but establishing correspondences between an image and a canonical 3D surface model in the wild remains an open challenge.

We introduce DensePose-COCO, a large-scale dataset for dense human pose estimation. Our annotation pipeline consists of two stages. In the first stage, annotators segment visible body parts into semantic regions (head, torso, upper arms, lower arms, upper legs, lower legs, hands, feet). In the second stage, for each body part, annotators identify corresponding surface points using six pre-rendered views of the SMPL body model. This two-stage decomposition reduces the cognitive load and enables efficient annotation. The resulting dataset contains over 5 million manually annotated correspondences across 50K persons.

We propose DensePose-RCNN, built upon Mask R-CNN with a Feature Pyramid Network (FPN) backbone. The architecture uses ROI-Align pooling to extract region features, followed by stacked 3x3 convolutional layers with 512 channels for the dense prediction head. For each detected person, the network predicts body part labels and UV coordinates within each part. We compare this region-based approach with a fully-convolutional variant (DP-FCN) and find that region-based processing provides significantly better accuracy, as it can focus computational resources on individual person instances.

A key challenge is that ground-truth annotations are spatially sparse — only a subset of surface points are annotated per person. To address this, we introduce a "teacher" inpainting network that completes the sparse annotations into dense correspondence maps. The teacher is trained on the annotated points and then used to generate dense pseudo-ground-truth for the full body surface. Training DensePose-RCNN with this distilled supervision yields substantial improvements: AUC at 10cm threshold increases from 0.315 to 0.381.

We evaluate on the DensePose-COCO benchmark using two metrics: Ratio of Correct Points (RCP) with area-under-curve measures, and Geodesic Point Similarity (GPS) inspired by the Object Keypoint Similarity metric. Our best model, DensePose-RCNN with cascading and distillation, achieves AUC of 0.390 at 10cm and 0.664 at 30cm. For comparison, human performance reaches 0.563 and 0.835 respectively. The fully-convolutional baseline (DP-FCN) only achieves 0.253 at 10cm, confirming the advantage of region-based processing. Per-instance evaluation using GPS yields AP of 55.8 with ResNet-50 backbone. The system operates at 20-26 fps on 240x320 images.

We have introduced DensePose, a system for establishing dense correspondences between RGB images and the 3D surface of the human body. Our contributions include the DensePose-COCO dataset with 50K annotated persons and 5M+ correspondence points, the DensePose-RCNN architecture that leverages region-based processing, and a teacher-student training scheme for handling sparse supervision. We have shown that dense human pose estimation is achievable at real-time speeds, opening up applications in augmented reality, graphics, and human-computer interaction.