In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. Grounding DINO achieves 52.5 AP on the COCO detection zero-shot transfer benchmark, setting a new record on the ODinW zero-shot benchmark with a mean 26.1 AP.

Object detection has long been studied under the closed-set setting, where the detector is trained and evaluated on a fixed set of object categories. This setting severely limits the applicability of detectors in real-world scenarios where novel objects appear frequently and cannot all be anticipated at training time. Open-set object detection addresses this limitation by allowing the detector to recognize objects from categories not seen during training, guided by natural language descriptions. Recent advances in vision-language pretraining have opened new possibilities for open-set detection, but existing approaches often struggle to achieve strong performance on both closed-set and open-set benchmarks simultaneously.

Our approach builds upon DINO, a state-of-the-art Transformer-based detector, and introduces tight cross-modality fusion at multiple stages. We divide the detection pipeline into three phases: feature extraction and enhancement, query initialization and selection, and box prediction and refinement. In the first phase, we employ a feature enhancer that performs bidirectional cross-attention between image and text features, allowing both modalities to be enriched by each other. Unlike approaches that only use language to query visual features, our bidirectional fusion allows visual context to also improve language understanding, leading to better grounding of ambiguous or context-dependent expressions.

In the second phase, we introduce language-guided query selection. Instead of using learnable queries that are fixed regardless of the input text, our approach selects the most relevant features from the enhanced image features based on their similarity to the text input. This ensures that the decoder focuses on image regions most likely to contain the objects described by the language input. In the third phase, a cross-modality decoder iteratively refines the predicted boxes while maintaining cross-attention between object queries and both image and text features, enabling precise localization guided by linguistic context.

We evaluate Grounding DINO on COCO, LVIS, ODinW, and RefCOCO/+/g benchmarks. On COCO zero-shot detection, Grounding DINO achieves 52.5 AP without any COCO training data, surpassing previous open-set detectors including GLIP (49.8 AP) and DetCLIP (45.8 AP). On the challenging LVIS benchmark with over 1,200 categories, our method demonstrates strong performance on rare categories (AP_r), achieving 32.7 compared to 26.9 for GLIP. For referring expression comprehension on RefCOCO, RefCOCO+, and RefCOCOg, Grounding DINO achieves state-of-the-art accuracy, demonstrating its versatility in understanding diverse linguistic inputs from simple category names to complex spatial descriptions.

We have presented Grounding DINO, an open-set object detector that effectively fuses language and vision through a three-phase tight fusion architecture. By marrying the powerful DINO detector with grounded pre-training, our model can detect arbitrary objects specified through natural language, achieving state-of-the-art results across multiple zero-shot, open-vocabulary, and referring expression benchmarks. Our work demonstrates that deep, multi-stage cross-modality fusion is key to bridging the gap between closed-set and open-set object detection.