We present a box-free bottom-up approach for the tasks of pose estimation and instance segmentation of people in multi-person images, using an efficient single-shot model. The proposed method, called PersonLab, detects individual keypoints and predicts their relative displacements, allowing us to group keypoints into person pose instances and semantically segment person regions, all without the need for a person detector.

Using a fully-convolutional neural network, PersonLab predicts (1) keypoint heatmaps for all keypoint types, (2) short-range offsets to refine keypoint locations, (3) mid-range pairwise offsets to associate keypoints belonging to the same person, and (4) person segmentation maps with long-range offsets for instance-level segmentation. We demonstrate state-of-the-art results for multi-person pose estimation and competitive results for instance segmentation on the COCO benchmark.

The standard approach to multi-person pose estimation follows a top-down paradigm: first detect people using an off-the-shelf person detector, then apply a single-person pose estimator to each detection box. While effective, this approach has several drawbacks: the runtime scales linearly with the number of people, errors in the person detector propagate to the pose estimation stage, and the person bounding boxes may overlap or be inaccurate.

In contrast, bottom-up approaches first detect all keypoints in the image and then group them into individual person instances. Our method, PersonLab, takes this approach further by jointly predicting keypoint detection, keypoint grouping, and instance segmentation in a single forward pass. The key innovation is the use of geometric embedding vectors that encode the spatial relationships between detected keypoints, enabling efficient and accurate person-instance assembly.

PersonLab employs a ResNet backbone to extract features, followed by prediction heads for each output type. For keypoint detection, the model produces K heatmaps, one for each keypoint type (e.g., nose, left shoulder, right knee). To improve localization precision, we additionally predict short-range offset vectors that point from each spatial position to the nearest keypoint of the corresponding type. The combination of heatmap detection with offset refinement allows sub-pixel localization accuracy while maintaining the efficiency of operating on a coarser feature map.

To associate detected keypoints belonging to the same person, we predict mid-range pairwise offset fields. For each pair of adjacent keypoint types in the kinematic tree (e.g., left shoulder to left elbow), the model predicts a 2D offset vector at each spatial position pointing from one keypoint type to the paired keypoint type. Given a detected keypoint, we use these offset fields to greedily assemble the remaining keypoints of the same person by following the kinematic chain. This approach is simpler than associative embedding approaches that require solving a complex grouping problem, and more geometrically interpretable.

We evaluate PersonLab on the COCO 2017 benchmark for both keypoint detection and instance segmentation. Using a ResNet-152 backbone with input resolution 1401, our model achieves 68.7 AP on the keypoint test-dev set, outperforming all previous bottom-up methods including CMU-Pose and Associative Embedding. The inference speed is approximately 200ms per image on a single GPU, which is significantly faster than top-down methods that need to run a pose estimator for each detected person.

For instance segmentation, PersonLab achieves 37.1 AP on the COCO person category, demonstrating that our unified framework can jointly handle both pose estimation and instance segmentation effectively. The long-range offset fields successfully assign pixels to their corresponding person instances. While dedicated instance segmentation methods like Mask R-CNN achieve higher segmentation accuracy, PersonLab offers the advantage of jointly solving both tasks in a single forward pass without requiring a region proposal network.

We have presented PersonLab, a unified bottom-up approach for multi-person pose estimation and instance segmentation. Our method uses a single-shot fully convolutional model to predict keypoint heatmaps, short/mid/long-range offset fields, enabling efficient person assembly and segmentation without relying on person detection. PersonLab achieves state-of-the-art results among bottom-up methods on COCO keypoint detection while simultaneously providing person instance segmentation.