We present an approach to efficiently detect the 2D pose of multiple people in an image. The approach uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to associate body parts with individuals in the image. The architecture encodes global context, allowing a greedy bottom-up parsing step that maintains high accuracy while achieving realtime performance, irrespective of the number of people in the image. The approach achieves first place in the inaugural COCO 2016 keypoints challenge and significantly surpasses the prior state-of-the-art result on the MPII Multi-Person benchmark.

Human 2D pose estimation — the problem of localizing anatomical keypoints or "parts" — has attracted significant attention. The focus of much previous work has been on single-person scenarios. Multi-person pose estimation presents distinct challenges: unknown number of people at varying positions and scales, complex spatial interference from contact and occlusion, and runtime complexity growing with the number of people. Existing approaches typically employ person detectors followed by single-person pose estimation, termed "top-down approaches". However, these suffer from "early commitment" — if the person detector fails, there is no recovery. Additionally, computational cost scales linearly with the number of detected people.

We contrast these with "bottom-up approaches", which first detect all body parts and then associate them to individuals. Bottom-up methods avoid early commitment and their computation does not grow with the number of people. However, previous bottom-up methods like DeepCut required hours of processing through integer linear programming, while DeeperCut needed several minutes per image. This work introduces Part Affinity Fields as a set of 2D vector fields that encode the location and orientation of limbs over the image domain. Through a greedy parsing algorithm that leverages PAFs' implicit global context, the method achieves realtime multi-person pose detection.

Multi-person pose estimation follows two paradigms. Top-down methods first apply a person detector and then estimate pose for each detection. These include Convolutional Pose Machines (CPM) and Stacked Hourglass Networks, which achieve high single-person accuracy but whose computational cost scales with the number of people. Bottom-up methods detect all body parts first, then group them. DeepCut formulated this as an NP-Hard integer linear program requiring hours to solve. DeeperCut improved speed with stronger part detectors but still required minutes per image. Our approach differs by encoding part associations in a continuous, differentiable representation (PAFs) that is jointly learned with part detection.

The architecture comprises two branches: one predicting detection confidence maps S, another predicting part affinity fields L. Each branch uses iterative prediction across multiple stages with intermediate supervision. Initial processing uses VGG-19 convolutional layers generating feature maps F. At stage one: S^1 = rho^1(F) and L^1 = phi^1(F). Subsequent stages refine predictions by concatenating prior outputs with original features: S^t = rho^t(F, S^{t-1}, L^{t-1}). Loss functions employ L2 distance weighted spatially to handle incomplete annotation, with intermediate supervision addressing vanishing gradients through periodic gradient replenishment.

PAFs represent limb location and orientation as 2D vector fields. For pixels on a limb, vectors point from one associated part toward another; elsewhere vectors are zero. The groundtruth definition states: L*_c,k(p) = v if p is on limb c,k; 0 otherwise, where v is the unit vector from one joint to its connected joint. Multiple people's affinity fields are averaged at overlapping limb regions. During testing, association confidence between body part candidates d_{j1} and d_{j2} is measured through a line integral over the PAF: E = integral from 0 to 1 of L_c(p(u)) dot (d_{j2} - d_{j1}) / ||d_{j2} - d_{j1}||_2 du. This line integral naturally measures how well the PAF agrees with the hypothesis that two detected parts belong to the same person.

Finding optimal part associations is an NP-Hard K-dimensional matching problem. We present a greedy relaxation using tree-structured skeleton instead of fully connected graphs, decomposing the problem into independent bipartite matching subproblems. For each limb type, maximum weight bipartite matching is applied, subject to constraints ensuring no shared nodes. Full-body poses follow from combining all limb assignments: max_Z E = sum from c=1 to C of max_{Z_c} E_c, solving each limb type independently. The greedy approach achieves comparable accuracy to global optimization at a fraction of the computational cost because PAFs implicitly encode global context through the large receptive fields of the multi-stage CNN.

On the MPII Multi-Person dataset, the method achieved 79.7% mean Average Precision on the 288-image subset, surpassing DeeperCut's 71.2% while processing in 0.005 seconds per image versus 230 seconds. On the full MPII test set, performance reached 75.6% mAP with multi-scale evaluation, compared to 59.5% for prior methods. Ablation studies demonstrated PAFs outperform midpoint representations by 2.9% in detection accuracy. On the COCO Keypoints Challenge, the method achieved 60.5% AP on the test-challenge set, placing first in the inaugural 2016 competition. Runtime analysis demonstrated that parsing complexity scales as O(n^2) for n people, yet parsing consumed negligible time compared to CNN processing.

We have presented realtime algorithms to detect the 2D pose of multiple people in images, enabling machines to interpret photograph significance and human behavior. The work presents an explicit nonparametric representation — Part Affinity Fields (PAFs) — that encodes both position and orientation of human limbs. Combined with confidence maps for body part detection and a greedy parsing algorithm, our method achieves state-of-the-art results while maintaining computational efficiency. The code has been publicly released to encourage reproducibility and future research. We believe bottom-up representations that jointly encode detection and association are a promising direction for multi-person pose estimation.