Video instance segmentation (VIS) aims to simultaneously detect, segment, and track object instances across video frames. Existing methods typically adopt a multi-stage pipeline: first performing per-frame detection and segmentation, then associating instances across frames via tracking. This paper proposes VisTR, the first end-to-end transformer-based framework for video instance segmentation. VisTR formulates the VIS task as a direct set prediction problem in the spatiotemporal domain, where instance predictions for all frames are generated simultaneously through a single forward pass. The approach achieves the highest speed among VIS models while maintaining competitive accuracy on the YouTube-VIS benchmark.

Video instance segmentation was introduced as a new task that unifies detection, segmentation, and tracking into a single evaluation framework. Current state-of-the-art methods follow a detect-then-track paradigm: they apply an image-level instance segmentation model (e.g., Mask R-CNN) to each frame independently, then use hand-crafted heuristics or separate tracking networks to associate detections across time. This multi-stage approach suffers from error propagation between stages, the need for complex post-processing (e.g., NMS, IoU-based matching), and inability to leverage temporal information during the detection phase.

Inspired by DETR's success in reformulating object detection as set prediction with transformers, we extend this paradigm to the video domain. VisTR treats the entire video clip as a single entity and directly outputs instance predictions for all frames simultaneously. The transformer's self-attention mechanism naturally captures spatiotemporal dependencies, allowing each instance query to attend to features across both space and time. This eliminates the need for separate tracking modules, NMS, and other post-processing heuristics.

MaskTrack R-CNN first established the VIS task and benchmark, extending Mask R-CNN with a tracking branch that uses learned embeddings for association. Subsequent works like SipMask, CompFeat, and STEm-Seg improved segmentation quality or temporal modeling but all maintained the multi-stage paradigm. In object detection, DETR demonstrated that transformers with set-based Hungarian matching can eliminate NMS and anchor generation. Deformable DETR further improved training efficiency. However, no prior work has applied the transformer-based set prediction framework to the video instance segmentation task.

VisTR processes a video clip of T frames simultaneously. Each frame is first processed by a CNN backbone (ResNet-50/101) to extract feature maps. The feature maps from all T frames are concatenated along the spatial dimension to form a single spatiotemporal feature sequence. This sequence is then fed into a Transformer encoder-decoder, where the encoder applies multi-head self-attention over the entire spatiotemporal sequence, enabling each spatial position in any frame to attend to all positions in all frames. The decoder takes N learned instance queries and produces N instance predictions, each containing class labels, bounding boxes, and segmentation masks for all T frames.

A key innovation is the instance sequence matching mechanism. Unlike DETR which matches predictions to per-image ground truth, VisTR performs matching between predicted instance sequences and ground truth instance sequences across all T frames using the Hungarian algorithm. The matching cost considers classification probability, bounding box L1 and GIoU losses, and mask dice loss, all accumulated across the T frames. This formulation naturally handles the tracking problem as a byproduct of the set prediction framework: each instance query is responsible for the same object across all frames, making explicit tracking unnecessary.

Evaluation is conducted on the YouTube-VIS 2019 benchmark, the standard dataset for VIS containing 2,238 training, 302 validation, and 343 test videos with 40 categories. With ResNet-50 backbone, VisTR achieves 36.2% AP on the validation set, comparable to MaskTrack R-CNN (30.3%) and SipMask (33.7%). With ResNet-101, performance improves to 40.1% AP. Critically, VisTR achieves the highest inference speed among all compared methods at 69.9 ms per clip (approximately 57.5 FPS for 36-frame clips), compared to MaskTrack R-CNN at 162.5 ms and STEm-Seg at 188.3 ms. The method demonstrates strong temporal consistency in segmentation masks without any post-processing.

Ablation experiments examine several design choices. Removing the instance sequence matching and using per-frame matching degrades AP by 3.1%, confirming the importance of temporal-aware matching. Reducing the number of encoder layers from 6 to 3 drops AP by 1.8%, indicating that sufficient self-attention depth is needed for spatiotemporal reasoning. The segmentation mask head using instance-level attention pooling outperforms simple feature concatenation by 2.4% AP. Increasing clip length from 18 to 36 frames improves AP by 1.2%, demonstrating the benefit of longer temporal context.

We present VisTR, the first end-to-end transformer-based model for video instance segmentation. By formulating VIS as direct set prediction in the spatiotemporal domain with instance sequence matching, VisTR eliminates the need for separate tracking modules, NMS, and other hand-designed post-processing. The approach achieves competitive accuracy with the highest inference speed on YouTube-VIS. Our work demonstrates that the transformer's ability to model long-range dependencies is particularly well-suited for tasks requiring joint spatiotemporal reasoning.