This work presents Sa2VA, the first unified model for dense grounded understanding of both images and videos. Unlike existing multi-modal large language models (MLLMs), which are often limited to specific modalities and tasks, Sa2VA supports a wide range of image and video tasks, including referring segmentation and conversation, with minimal one-shot instruction tuning. The model combines SAM-2 with LLaVA and unifies text, image, and video in a shared LLM token space. The authors introduce the Ref-SAV dataset with 72k+ object expressions and achieve state-of-the-art results across multiple benchmarks.

Multi-modal Large Language Models (MLLMs) have made significant progress with visual question answering, narrative generation, and interactive editing. However, there exists a fundamental gap: video perception models like SAM-2 lack open-ended understanding abilities, while video MLLMs cannot perform pixel-level perception tasks like segmentation. The authors observe that "no work can leverage the benefits from both sides." Three key challenges emerge: (1) task formulation for multimodal inputs; (2) balancing performance across diverse tasks; and (3) effective knowledge sharing from pretrained models.

Sa2VA addresses these challenges with a decoupled design. Rather than deeply integrating SAM-2 into the LLM, the framework uses special [SEG] tokens as bridges: the hidden states of the [SEG] token generated by the LLM are used as spatial-temporal prompts and fed into SAM-2's decoder. This design maintains the plug-and-play compatibility — SAM-2 and LLaVA operate as independent modules connected through a lightweight interface. For video tasks, key frames generate masks via [SEG] tokens while remaining frames use SAM-2's memory mechanisms for tracking.

Multi-modal LLMs have evolved from image-level understanding (LLaVA, InternVL) to video handling (LLaVA-OneVision), but lack pixel-level grounding capabilities. Referring segmentation methods have progressed from fusion modules to transformer-based approaches to LLM-equipped architectures, yet the conceptual vocabulary remains limited compared with the knowledge space of LLMs. Video segmentation and grounding methods focus on closed-set pixel segmentation with predefined categories. Existing approaches either sacrifice language understanding for segmentation precision or vice versa. Sa2VA is the first to unify both capabilities without such tradeoffs.

Sa2VA reformulates diverse tasks under a unified representation. Referring image/video object segmentation takes text tokens and images/videos as input, outputting binary masks or spatio-temporal masklets. Image/video chat and grounded caption generation outputs answer text with aligned masks. Visual prompt understanding accepts additional visual prompts (boxes/points). All tasks are unified as: T_o, M_o = LLM({I_i, V_i, VP_i}, T_i), where outputs are text and/or masks depending on task type. The LLM generates [SEG] tokens whose hidden states serve as spatial-temporal prompts for SAM-2's decoder. The training loss combines text regression loss with pixel-wise cross-entropy and Dice loss for segmentation.

The authors introduce Ref-SAV, a dataset built from 37,311 videos with 72,509 object expressions from the SA-V dataset. The annotation pipeline has three stages: (1) object/part-level annotation using InternVL2-76B with consistency checking; (2) scene-level annotation with object relationships; and (3) video-level annotation capturing motion and actions across 8 uniformly sampled frames. The benchmark subset contains 1,147 videos with 1,945 object expressions (1,694 long, 251 short), manually validated. Ref-SAV features long text descriptions, heavy occlusion, and large motion — challenges absent from existing benchmarks.

On image referring segmentation, Sa2VA-8B achieves 81.6, 76.2, and 78.7 cIoU on RefCOCO, RefCOCO+, and RefCOCOg respectively, surpassing prior state-of-the-art. On video referring segmentation, Sa2VA achieves 57.0, 75.2, and 57.6 J&F on MeVIS, Ref-DAVIS17, and ReVOS, surpassing the previous SOTA VISA-13B by 12.5, 4.8, and 6.7 J&F. For chat performance, Sa2VA maintains strong results on MME (1651/578), MMBench (82.4), and SEED-Bench (75.5) while retaining grounding capabilities. On the new Ref-SAV benchmark, Sa2VA-8B achieves 49.3 J&F zero-shot vs. 14.1 for prior methods, improving to 58.7 with training data. Ablation studies confirm that co-training across all tasks benefits each individual task, and that the single [SEG] token design outperforms multi-token variants.

Sa2VA presents a versatile framework that integrates SAM-2 with LLaVA-like MLLMs to achieve dense, grounded understanding of both images and video. The decoupled design with [SEG] tokens enables plug-and-play integration while maintaining state-of-the-art performance across segmentation, conversation, and grounded captioning tasks. The Ref-SAV benchmark provides a challenging evaluation platform for future research. The model handles multiple tasks with a single instruction-tuning process, demonstrating that perception and understanding need not be separate capabilities in multi-modal AI.