This paper presents PointRend (Point-based Rendering), a method that "performs point-based segmentation predictions at adaptively selected locations based on an iterative subdivision algorithm." The key insight draws an analogy between image segmentation and image rendering in computer graphics: both problems involve mapping a representation to a regularly-sampled grid of pixel labels or colors. PointRend produces "crisp object boundaries in regions that are over-smoothed by previous methods" and works with both instance segmentation and semantic segmentation.

Current segmentation architectures operate on regular grids — either through fully convolutional networks that predict dense pixel-wise labels or through mask heads operating on fixed-resolution feature maps. These approaches face a fundamental tension: "computation is uniformly allocated across the grid, even though only a small fraction of grid cells differ from their neighbors." The vast majority of pixels lie in smooth interior regions where prediction is trivial.

The authors observe that this inefficiency parallels a long-solved problem in computer graphics: "efficient rendering of high-resolution images does not require estimating a color value for every pixel." Techniques like adaptive subdivision and anti-aliasing in image rendering focus computation on regions where the image signal changes rapidly, such as object edges. PointRend borrows this principle: compute predictions only at carefully selected points where uncertainty is highest.

Prior work on improving segmentation resolution includes dilated/atrous convolutions that maintain spatial resolution, encoder-decoder architectures that progressively upsample, and conditional random fields (CRFs) used as post-processing to sharpen boundaries. In instance segmentation, Mask R-CNN predicts masks on a "coarse 28x28 grid" which is then "resized to the bounding box resolution, resulting in over-smoothed boundaries." These approaches uniformly process all spatial locations regardless of their difficulty.

The PointRend module operates as a drop-in replacement that can be applied to existing segmentation architectures. For each selected point, the module constructs a point-wise feature representation by combining two types of features: (1) fine-grained features extracted via bilinear interpolation from the CNN feature map at the point's location, capturing low-level positional information; and (2) coarse-grained features from the region's global representation, providing semantic context. A small point head MLP then maps this combined representation to a segmentation label.

During training, PointRend uses a non-uniform sampling strategy biased toward uncertain regions. Points are selected by first uniformly sampling candidates, then oversampling points with predictions closest to 0.5 (most uncertain). During inference, the module employs an iterative subdivision algorithm inspired by adaptive rendering: starting from a coarse prediction, it "selects the N most uncertain points, computes their representations, and predicts labels" at each subdivision step, progressively refining only the ambiguous regions until the desired output resolution is reached.

Experiments demonstrate PointRend's effectiveness across both instance and semantic segmentation. For instance segmentation on COCO, PointRend applied to Mask R-CNN with a ResNet-50 backbone achieves significant improvements in mask AP, particularly visible at higher IoU thresholds where boundary quality matters most. Qualitatively, predictions show dramatically sharper boundaries compared to the standard 28x28 mask head. For semantic segmentation on Cityscapes, PointRend improves results with reduced computational cost compared to increasing feature map resolution. The method adds negligible computational overhead since it processes only a small subset of all pixels.

PointRend presents a new perspective on image segmentation by drawing an analogy to rendering in computer graphics. The module adaptively selects a non-uniform set of points at which to compute segmentation labels, using an iterative subdivision strategy that focuses computation on uncertain regions. This approach yields significantly sharper boundaries while being computationally efficient and applicable as a general module across different segmentation architectures and tasks.