We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection.

The vision community has rapidly improved object detection and semantic segmentation results over a short period of time. In large part, these advances have been driven by powerful baseline systems, such as, respectively, Fast/Faster R-CNN and Fully Convolutional Network (FCN) frameworks. These methods are conceptually intuitive and offer flexibility and robustness, together with fast training and inference time. Our goal in this work is to develop a comparably enabling framework for instance segmentation. Instance segmentation is challenging because it requires the correct detection of all objects in an image while also precisely segmenting each instance. It therefore combines elements from the classical computer vision tasks of object detection and semantic segmentation.

Our approach, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks on each Region of Interest (RoI), in parallel with the existing branch for classification and bounding box regression. The mask branch is a small FCN applied to each RoI, predicting a segmentation mask in a pixel-to-pixel manner. Mask R-CNN is simple to implement and train given the Faster R-CNN framework, which facilitates a wide range of flexible architecture designs. Additionally, the mask branch only adds a small computational overhead, enabling a fast system and rapid experimentation. In principle, Mask R-CNN is an intuitive extension of Faster R-CNN, yet constructing the mask branch properly is critical for good results. Most importantly, Faster R-CNN was not designed for pixel-to-pixel alignment between network inputs and outputs. This is most evident in how RoIPool, the de facto core operation for attending to instances, performs coarse spatial quantization for feature extraction. To fix the misalignment, we propose a simple, quantization-free layer, called RoIAlign, that faithfully preserves exact spatial locations.

Instance segmentation has been addressed by prior methods following either a segment-then-recognize strategy or a detect-then-segment approach. The former generates category-agnostic segments and then classifies them. Methods like MNC and FCIS represent the detect-then-segment paradigm. FCIS introduced position-sensitive score maps that are shared across detection and segmentation, but exhibited systematic artifacts on overlapping instances, requiring complex post-processing. In contrast, our method takes the instance-first strategy, predicting masks within detected bounding boxes, which naturally handles instance overlap and avoids the need for cascaded stages or specialized architectures.

Mask R-CNN is conceptually simple: Faster R-CNN has two outputs for each candidate object, a class label and a bounding-box offset; to this we add a third branch that outputs the object mask. The additional mask output is distinct from the class and box outputs, requiring extraction of much finer spatial layout of an object. The training loss is defined as the multi-task loss L = L_cls + L_box + L_mask. The mask branch generates a K x m x m dimensional output for each RoI, encoding K binary masks at m x m resolution, one for each of the K classes. We apply a per-pixel sigmoid and define the mask loss as the average binary cross-entropy loss. This allows the network to generate masks for every class without competition among classes, decoupling mask and class prediction.

RoIPool is a standard operation for extracting a small feature map from each RoI. RoIPool first quantizes a floating-number RoI to the discrete granularity of the feature map, this quantized RoI is then subdivided into spatial bins which are themselves quantized, and finally feature values covered by each bin are aggregated (usually by max pooling). These quantizations introduce misalignments between the RoI and the extracted features. While this may not impact classification which is robust to small translations, it has a large negative effect on predicting pixel-accurate masks. To address this, we propose RoIAlign, which avoids any quantization of the RoI boundaries or bins, instead using bilinear interpolation to compute the exact values of the input features at four regularly sampled locations in each RoI bin, and then aggregating the result. RoIAlign leads to large improvements of 10% to 50% (relative) on mask AP, showing its critical importance.

We instantiate Mask R-CNN with multiple architectures. We use the term backbone to refer to the feature extraction network, and head to denote the task-specific sub-network applied to each RoI. We evaluate with ResNet and ResNeXt networks of depth 50 or 101 layers, combined with Feature Pyramid Networks (FPN). The mask head is a fully convolutional network (FCN) that takes the RoI features and predicts an m x m mask through a series of convolutional layers. Compared to an alternative MLP-based approach that collapses spatial structure, the FCN approach preserves explicit spatial layout and improves mask AP by 2.1 points. For the FPN backbone, we use a lighter head design: four consecutive 3x3 convolutions with 256 channels, followed by a 2x2 deconvolution for upsampling.

We perform a comprehensive evaluation on the COCO dataset. We train on the trainval35k split (80k training + 35k validation subset) and evaluate on minival and test-dev. Our main results with ResNet-101-FPN backbone achieve 35.7 mask AP, and with ResNeXt-101-FPN achieve 37.1 mask AP, outperforming FCIS+++, the winner of the 2016 COCO segmentation challenge. In ablation studies, RoIAlign improves mask AP by approximately 3 points at stride 16, and 7.3 points at stride 32 compared to RoIPool. The sigmoid-based mask prediction outperforms softmax by 5.5 AP. For object detection, Mask R-CNN achieves 38.2 box AP with ResNet-101-FPN, with the multi-task training alone contributing a 0.9 point gain over detection-only training.

We show that the Mask R-CNN framework can be extended to human pose estimation with minimal modification. We model keypoint detection as a one-hot binary mask prediction problem: for each of the K keypoints, a one-hot m x m binary mask with only a single pixel labeled as foreground is predicted. The keypoint head consists of a stack of eight 3x3 512-d convolutional layers, followed by a deconvolution layer and bilinear upsampling. Our model achieves 62.7 keypoint AP on COCO test-dev, surpassing the winner of the 2016 COCO keypoint detection challenge. A unified model that simultaneously predicts boxes, segments, and keypoints runs at approximately 5 fps, demonstrating the versatility of the Mask R-CNN framework.

We presented Mask R-CNN, a simple yet effective framework for instance segmentation that also enables human pose estimation. The key elements include RoIAlign for preserving spatial information, and decoupled mask and class prediction using per-class binary masks. Our method achieves state-of-the-art results on COCO across three challenging tasks: instance segmentation, object detection, and keypoint detection. Despite its simplicity, Mask R-CNN outperforms all existing, heavily-engineered entries in every track. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition.