This paper proposes DeepCap, a novel deep learning approach for monocular dense human performance capture. The method is trained in a weakly supervised manner based on multi-view supervision, removing the need for 3D ground truth annotations. The architecture disentangles the task into pose estimation and non-rigid surface deformation, enabling dense tracking of human body shape and motion from a single RGB camera.

Human performance capture — reconstructing dense 3D human geometry and motion — is fundamental to applications in visual effects, virtual reality, and telepresence. Existing approaches either require expensive multi-camera setups at test time or rely on 3D ground truth data that is extremely difficult to obtain at scale. The central challenge is: can we achieve dense performance capture from a single monocular camera without requiring 3D supervision?

Recent learning-based methods for human pose estimation (e.g., HMR, SPIN) focus on parametric body model fitting but capture only coarse body shape without surface details like clothing deformation. Optimization-based methods can recover fine details but are slow and require careful initialization. DeepCap bridges this gap through a learning-based approach that captures both skeletal pose and non-rigid surface deformation.

Multi-view performance capture systems use multiple synchronized cameras to reconstruct dense geometry via multi-view stereo or volumetric fusion, achieving high quality but limiting deployment to controlled studio environments. Template-based tracking methods deform a pre-scanned mesh to match observations using optimization-based energy minimization. Monocular approaches using SMPL or SCAPE body models predict pose parameters but cannot model non-rigid deformations beyond the parametric model space.

The first stage predicts skeletal pose parameters from a single RGB image. A convolutional encoder extracts image features and regresses joint angles and global translation of a predefined skeleton. Training uses multi-view consistency as supervision: the predicted 3D pose is projected onto each camera view, and the reprojection error across all views serves as the loss function. This eliminates the need for 3D pose ground truth annotations, using only 2D keypoint detections from multi-view images.

The second stage predicts per-vertex displacements on top of the posed body template to capture clothing deformation and surface details. Given the pose-driven mesh from stage one, a graph convolutional network processes the mesh structure combined with image features projected onto mesh vertices. Training supervision comes from multi-view silhouette consistency and photometric loss: the deformed mesh is rendered from each view and compared against observed silhouettes and RGB images.

Experiments are conducted on multi-view studio recordings of multiple subjects performing various motions. Quantitative evaluation uses per-vertex error measured against multi-view reconstruction ground truth. DeepCap achieves mean vertex error of approximately 2-3 cm, significantly outperforming HMR and optimization-based baselines. The method runs at near real-time speed, whereas optimization-based approaches require minutes per frame. Ablation studies confirm that both the disentangled architecture and multi-view supervision are critical — removing either significantly degrades performance.

DeepCap presents the first weakly supervised deep learning method for monocular dense human performance capture. By disentangling pose estimation and non-rigid deformation and leveraging multi-view supervision during training, the method achieves accurate dense reconstruction from a single camera at test time without requiring any 3D ground truth annotations. The approach opens new possibilities for accessible human performance capture outside controlled studio environments.