We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN), introducing a novel architecture tailored for SDS.

Traditional object detection outputs a bounding box for each detected object, which provides only a coarse localization. Semantic segmentation assigns each pixel a class label but does not distinguish between different instances of the same class. The SDS task bridges these two by requiring instance-level segmentation: each detected object must be accompanied by a pixel-level mask. This is critical for applications like robotic manipulation, autonomous driving, and image editing.

Our approach has two stages. First, we use MCG (Multiscale Combinatorial Grouping) to generate candidate region proposals, and classify them using CNN features extracted from both the bounding box region and the foreground region. The concatenation of these two feature types provides complementary information: the bounding box features capture context while the foreground features capture the object's appearance.

After initial detection, we refine the segmentation using category-specific, top-down figure-ground predictions. For each detected region, we train a CNN to predict which pixels belong to the object (figure) versus background (ground). This refinement step is critical: it improves segmentation quality by 7 points in the SDS metric, a 16% relative improvement over the baseline.

We evaluate on PASCAL VOC 2012 using the SDS metric (AP at IoU threshold of 0.5 measured on segmentations). Our method achieves state-of-the-art results on both the SDS task and the classical semantic segmentation task. On SDS, we obtain a mean AP of 49.7%. On semantic segmentation, our approach provides a 5 point boost (10% relative) over the previous state-of-the-art. On object detection with bounding boxes, we also achieve competitive performance.

We have introduced the Simultaneous Detection and Segmentation (SDS) task and proposed an effective approach combining region-based CNN features with top-down figure-ground refinement. Our results demonstrate that jointly addressing detection and segmentation leads to better performance on both tasks. We believe SDS will become an important task as the community moves beyond bounding boxes toward pixel-level understanding.