We propose an approach for semi-automatic annotation of object instances. We frame the object segmentation task as a polygon prediction problem, where the model takes as input an image crop and sequentially produces vertices of the polygon outlining the object. This allows the annotator to interfere at any time and correct a vertex, upon which the model updates the prediction. We show that our approach achieves "a factor of 4.7" speed improvement across Cityscapes classes with "78.4% agreement in IoU with original ground-truth," matching typical inter-annotator agreement levels. For vehicles specifically, the speed-up reaches 7.3x with 82.2% agreement.

Semantic segmentation demands large-scale annotated datasets. While deep learning approaches achieve impressive results, they remain "data hungry" with performance correlated to training data volume. Most large-scale segmentation datasets such as Cityscapes employed polygon-based annotation by human annotators — drawing polygon vertices around objects. This process is extremely time-consuming and expensive. The authors propose Polygon-RNN, which generates polygon vertices sequentially from image crops within bounding boxes, allowing human corrective intervention while maintaining structural coherence.

Existing semi-automatic annotation approaches include scribble-based methods, GrabCut variants, and superpixel labeling. These pixel-level graphical models struggle incorporating shape priors and often produce holes or require extensive manual correction. The authors position their polygon approach as more naturally matching existing annotation practices — most datasets are annotated with polygons, not pixel masks. The structured output (polygon) is also easier for human correction: adjusting a vertex is more intuitive than fixing scattered pixel labels.

The model architecture comprises two components. The CNN image encoder adapts VGG-16, removing fully connected layers and using skip-connections to concatenate multi-scale features, achieving 8x downsampling relative to the input. The RNN vertex predictor employs a two-layer Convolutional LSTM with 3x3 kernels and 16 channels. At each timestep, the ConvLSTM receives concatenated input including CNN features, one-hot encodings of the previous two vertices, and the first vertex encoding. The output represents a DxD grid (D=28) plus an end-of-sequence token through one-hot encoding, predicting the next vertex position on a quantized grid.

Training uses cross-entropy loss at each RNN timestep with smoothed target distributions — non-zero probability mass is assigned to locations within distance 2 in the output grid, softening the hard one-hot targets. The Adam optimizer trains with batch size 8 and learning rate 1e-4, decayed by 10x after 10 epochs. First vertex prediction uses a multi-task loss combining logistic losses for boundaries and vertices. Data augmentation includes random flipping, context expansion (10-20% beyond bounding box), and random polygon starting point selection. Training completes in approximately one day on an Nvidia Titan-X GPU.

During inference, the model selects highest log-probability vertices at each timestep. The key innovation is the annotator-in-the-loop mechanism: annotators can correct predicted vertices by inputting corrections at any step, which then feeds into subsequent predictions. This means each human correction propagates forward, improving all subsequent vertices. Typical inference requires only 250 milliseconds per object, enabling real-time interactive annotation. Simulated annotator experiments with threshold T (correcting predictions deviating beyond distance T) show that at T=3, only 9.39 clicks per instance achieves 78.40% IoU agreement.

On Cityscapes without annotator interaction, Polygon-RNN outperforms DeepMask, SharpMask, and Dilation10 on six of eight categories. For cars, it achieves 71.17% IoU, exceeding SharpMask by 6%. In human evaluation on 101 car instances, the model reaches 82% IoU agreement with only 4.6 average clicks (7.3x speed-up) and 87.7% with 9.3 clicks (3.6x speed-up). Compared to GrabCut, which requires 17.5 clicks per instance, Polygon-RNN needs only 5.0 to 9.6 clicks. On KITTI (741 instances, cross-dataset generalization), the model achieves comparable IoU to human agreement with 5.84 clicks per instance.

Polygon-RNN provides a new paradigm for object instance annotation by framing segmentation as sequential polygon vertex prediction. The method produces structurally plausible polygon representations, enables user correction for desired accuracy levels through minimal clicks, and generalizes across datasets. Achieving a "speed-up of factor 4.74" while matching inter-annotator agreement demonstrates the practical utility for building large-scale annotation benchmarks.