The paper introduces HQTrack, a framework combining video multi-object segmentation (VMOS) with mask refinement (MR). The system addresses object tracking challenges by propagating masks across frames and applying pre-trained models to enhance mask quality. The authors achieved second place in the VOTS2023 challenge without using test-time augmentations or model ensembles, demonstrating the effectiveness of the two-component architecture in high-quality tracking and segmentation.

Visual object tracking represents a foundational task in computer vision with applications spanning robotics and autonomous driving. The VOTS2023 challenge broadens traditional tracking constraints by merging short-term and long-term tracking, single and multi-target scenarios, using segmentation masks as the primary localization method. This integration presents distinct challenges including inter-object relationship understanding, multi-target trajectory maintenance, and accurate mask estimation. Previous approaches divide into online-update trackers and Siamese-based methods, while recent Transformer-based architectures offer superior long-range modeling but primarily address single-object tracking with bounding box outputs.

VOTS2023 presents three primary challenges: (i) Extended video sequences exceeding 10,000 frames demand robust appearance discrimination and environmental adaptation; (ii) Objects disappearing and reappearing require specialized tracking mechanisms; (iii) Fast motion, occlusion, distractors, and tiny objects compound difficulties. The authors propose HQTrack, comprising a video multi-object segmenter (VMOS) built upon an improved DeAOT variant with cascaded gated propagation modules and InternImage-T feature extraction, and a mask refiner using HQ-SAM to address SAM's limitations with complex object structures.

DeAOT incorporates identification mechanisms for unified multi-object association within shared embedding spaces. Its hierarchical propagation through dual-branch gated propagation modules decouples visual and identification embedding propagation, mitigating object-agnostic visual information loss during deep propagation. VMOS extends DeAOT by cascading 1/8 scale gated propagation modules for improved tiny object perception. Multi-scale propagation features feed into FPN-based decoders alongside encoder features. InternImage-T replaces standard backbones, leveraging deformable convolutions for enhanced object discrimination across representative vision tasks. A fixed-length long-term memory of 8 frames addresses memory constraints in lengthy sequences.

SAM demonstrates powerful zero-shot capabilities through large-scale training on 1.1 billion masks but struggles with intricate object structures. HQ-SAM addresses this by injecting learnable output tokens into SAM's mask decoder while maintaining promptable design, efficiency, and zero-shot generalizability. HQTrack employs HQ-SAM as its mask refiner, extracting bounding boxes from VMOS predictions as prompts for HQ-SAM. An IoU threshold selector prevents harmful refinement: when IoU between VMOS and HQ-SAM masks exceeds the threshold, refined masks replace originals; otherwise original masks persist. This selective strategy ensures refinement only when beneficial.

InternImage-T serves as the encoder backbone, balancing accuracy and efficiency. Training occurs in two stages: first-stage pre-training uses synthetic sequences from static image datasets; second-stage fine-tuning employs multi-object segmentation datasets including DAVIS, YouTubeVOS, VIPSeg, BURST, MOTS, and OVIS for robust multi-object understanding. Training utilized two A100 GPUs with batch size 16. The long-term memory gap is set to 50 frames (optimized for VOTS video lengths), and inference follows the pipeline without test-time augmentation, multi-scale testing, or model ensembles.

Component-wise VMOS analysis demonstrated that replacing ResNet50 with InternImage-T improved AUC to 0.611. Adding multi-scale propagation mechanisms achieved 0.650 AUC, representing 3.9% improvement. Long-term memory gap analysis identified 50 frames as optimal for VOTS video sequences. For mask refinement, selective refinement using 0.1 IoU threshold optimally balanced benefits against degradation. Final VOTS2023 test set results: InternImage-T replacement yielded 3.2% AUC increase; SAM-H refinement added 1.4%; HQ-SAM-H boosted final AUC to 0.615, ranking second. Joint tracking paradigm showed superiority over separate tracking, attributed to improved inter-target relationship understanding.

HQTrack demonstrates powerful object tracking and segmentation capabilities through its two-component architecture. VMOS manages multi-target propagation across video frames while the mask refiner enhances segmentation quality. The framework achieved second place in VOTS2023 without auxiliary techniques, validating its effectiveness for high-quality tracking across diverse scenarios including long-term sequences, object disappearance/reappearance, and challenging visual conditions.