Most methods for object instance segmentation require all training examples to be labeled with segmentation masks. This paper proposes a partially supervised training paradigm that enables learning instance segmentation on categories that have only bounding box annotations, by leveraging mask annotations available for a subset of categories. The core idea is a weight transfer function that predicts mask prediction parameters from detection parameters, enabling the model to generalise segmentation capability to novel categories. The authors train Mask R-CNN with this approach on 3000 visual concepts from Visual Genome using COCO mask annotations for only 80 categories, achieving significant improvements over baselines.

Instance segmentation — detecting objects and predicting their pixel-level masks — is a fundamental computer vision task. Current state-of-the-art methods like Mask R-CNN require densely annotated segmentation masks for all categories. However, obtaining mask annotations is significantly more expensive than bounding box annotations — typically 5-10 times more costly per instance. This limits instance segmentation systems to a small number of categories (typically about 100 in COCO's 80 categories).

The authors ask: "Is it possible to train high-quality instance segmentation models without complete instance segmentation annotations for all categories?" They propose a partially supervised setting where the category set C splits into set A (with both box and mask annotations) and set B (with only box annotations). The key hypothesis is that detection weights encode visual appearance information that can be transferred to predict segmentation weights for categories in set B, without ever seeing their masks during training.

Instance segmentation methods, particularly Mask R-CNN, have achieved impressive results by extending Faster R-CNN with a mask prediction branch. Weight prediction networks (or hypernetworks) generate the parameters of one network from the output of another, enabling dynamic architectures. Transfer learning approaches leverage knowledge from data-rich domains to improve performance in data-scarce settings. Weakly supervised segmentation methods use image-level labels or bounding boxes as supervision but typically lag behind fully supervised approaches in quality. The class-agnostic mask prediction approach provides a natural baseline — training a single mask predictor shared across all categories — but ignores class-specific shape priors.

The method builds on Mask R-CNN, which predicts a class-specific binary mask for each detected region of interest. In standard Mask R-CNN, the mask head has class-specific parameters w_seg^c for each category c, which are directly learned from mask annotations. The authors propose replacing this direct learning with a weight transfer function that computes w_seg^c = T(w_det^c; theta), where w_det^c are the detection classification weights for category c and T is a learnable neural network parameterised by theta.

The weight transfer function T is implemented as a small fully-connected neural network. During training, for categories in set A (with masks), both the detection and segmentation losses are active. For categories in set B (box-only), only the detection loss is used, but the detection weights w_det^c are still updated. At test time, T generates mask parameters for all categories, including those never seen with mask annotations. An important training detail is gradient stopping on the detection weights — preventing gradients from the mask loss from modifying w_det^c — to maintain homogeneity of the class embedding space between sets A and B.

The authors further propose fusing FCN and MLP mask heads to capture complementary information. The MLP mask predictor captures the "gist" of an object's shape (coarse outline), while the FCN mask predictor captures fine details (edges, concavities). This fusion improves segmentation quality for both seen and unseen categories. The complete model is termed MaskX R-CNN — Mask R-CNN extended to handle partially supervised training.

Evaluation on COCO 80 categories split into VOC (20 classes) and non-VOC (60 classes) subsets demonstrates 40% relative improvement in mask AP for unseen categories compared to the class-agnostic baseline. Ablation studies examine the impact of input embeddings, transfer function architecture, and training procedures. On the large-scale Visual Genome experiment (3000 categories), the model produces reasonable segmentation masks using only 80 COCO mask annotations, with qualitative results showing coherent masks even for abstract concepts like "shadows" and "paths". The end-to-end training with gradient stopping outperforms stage-wise training approaches.

This paper addresses a fundamental bottleneck in instance segmentation: the requirement for mask annotations for all categories. The proposed weight transfer function enables partially supervised training where mask predictions can be made for categories seen only with bounding boxes. By leveraging the insight that detection weights serve as class embeddings encoding visual appearance, the approach achieves significant improvements over baselines and scales to thousands of categories. The framework opens the door to practical large-vocabulary instance segmentation systems that can leverage the abundantly available bounding box datasets.