The You Only Look Once (YOLO) series of detectors have established themselves as efficient and practical tools. However, their reliance on predefined and trained object categories limits their applicability in open scenarios. Addressing this limitation, we introduce YOLO-World, an innovative approach that enhances YOLO with open-vocabulary detection capabilities through vision-language modeling and pre-training on large-scale datasets. Specifically, we propose a new Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate the interaction between visual and linguistic information. Our method excels in detecting a wide range of objects in a zero-shot manner with high efficiency. On the challenging LVIS dataset, YOLO-World achieves 35.4 AP with 52.0 FPS on V100, which outperforms many state-of-the-art methods in terms of both accuracy and speed. Furthermore, the fine-tuned YOLO-World achieves remarkable performance on several downstream tasks, including object detection and open-vocabulary instance segmentation.

Object detection has been a long-standing and fundamental challenge in computer vision with numerous applications in image understanding, robotics, and autonomous vehicles. Tremendous works have achieved significant breakthroughs with the development of deep neural networks. Despite the success of these methods, they remain limited as they only handle object detection with a fixed vocabulary, for example, 80 categories in the COCO dataset. Once object categories are defined and labeled, trained detectors can only detect those specific categories, thus limiting the ability and applicability of open scenarios.

Recent works have explored the prevalent vision-language models to address open-vocabulary detection through distilling vocabulary knowledge from language encoders, such as BERT. However, these distillation-based methods are much limited due to the scarcity of training data with a limited diversity of vocabulary, for example, OV-COCO containing 48 base categories. Several methods reformulate object detection training as region-level vision-language pre-training and train open-vocabulary object detectors at scale. However, those methods still struggle for detection in real-world scenarios, which suffer from two aspects: (1) heavy computation burden and (2) complicated deployment for edge devices. Previous works have demonstrated the promising performance of pre-training large detectors while pre-training small detectors to endow them with open recognition capabilities remains unexplored.

In this paper, we present YOLO-World, aiming for high-efficiency open-vocabulary object detection, and explore large-scale pre-training schemes to boost the traditional YOLO detectors to a new open-vocabulary world. Specifically, YOLO-World follows the standard YOLO architecture and leverages the pre-trained CLIP text encoder to encode the input texts. We further propose the Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN) to connect text features and image features for better visual-semantic representation. During inference, the text encoder can be removed and the text embeddings can be re-parameterized into weights of RepVL-PAN for efficient deployment. We further investigate the open-vocabulary pre-training scheme for YOLO detectors through region-text contrastive learning on large-scale datasets, which unifies detection data, grounding data, and image-text data into region-text pairs. We also explore a prompt-then-detect paradigm to further improve the efficiency of open-vocabulary object detection in real-world scenarios.

Prevalent object detection research concentrates on fixed-vocabulary (close-set) detection, in which object detectors are trained on datasets with pre-defined categories, such as COCO dataset and Objects365 dataset, and then detect objects within the fixed set of categories. The methods for traditional object detection can be categorized into three groups: region-based methods (e.g., Faster R-CNN), pixel-based methods (one-stage detectors), and query-based methods (e.g., DETR). In terms of inference speed, YOLOs exploit simple convolutional architectures for real-time object detection. Several works propose various designs including path aggregation networks, cross-stage partial networks, and re-parameterization, which further improve both speed and accuracy.

Open-vocabulary object detection (OVD) has emerged as a new trend, aiming to detect objects beyond the predefined categories. Early works follow the standard OVD setting by training detectors on base classes and evaluating the novel (unknown) classes, but this is still limited for open scenarios and lacks generalization ability. Inspired by vision-language pre-training, recent works formulate OVD as image-text matching and exploit large-scale image-text data. GLIP presents a pre-training framework based on phrase grounding; Grounding DINO incorporates grounded pre-training into detection transformers. However, these methods often use heavy detectors like ATSS or DINO with Swin-L as backbone, leading to high computational demands and deployment challenges.

The traditional object detection methods, including the YOLO-series, are trained with instance annotations consisting of bounding boxes and category labels. We reformulate the instance annotations as region-text pairs, where the text corresponds to the region. Specifically, the text can be the category name, noun phrases, or object descriptions. Moreover, YOLO-World adopts both the image and texts (a set of nouns) as input and outputs predicted boxes and the corresponding object embeddings.

The overall architecture of YOLO-World consists of a YOLO detector, a Text Encoder, and a Re-parameterizable Vision-Language Path Aggregation Network (RepVL-PAN). The YOLO detector is based on YOLOv8, containing a Darknet backbone as the image encoder, a path aggregation network (PAN) for multi-scale feature pyramids, and a head for bounding box regression and object embeddings. The CLIP text encoder is adopted to extract text embeddings, offering better visual-semantic capabilities compared to text-only language encoders. The paper presents a text contrastive head to obtain the object-text similarity by computing the dot product between normalized object embeddings and normalized text embeddings, with learnable scaling and shifting factors.

Training with Online Vocabulary. During training, we construct an online vocabulary for each mosaic sample containing 4 images. Specifically, we sample all positive nouns involved in the mosaic images and randomly sample some negative nouns from the corresponding dataset. The vocabulary for each mosaic sample contains at most M nouns, and M is set to 80 as default.

Inference with Offline Vocabulary. At the inference stage, we present a prompt-then-detect strategy with an offline vocabulary for further efficiency. The user can define a series of custom prompts, which might include captions or categories. The text encoder is utilized to encode these prompts and obtain offline vocabulary embeddings. The offline vocabulary allows for avoiding computation for each input and provides the flexibility to adjust the vocabulary as needed.

The structure of the proposed RepVL-PAN follows the top-down and bottom-up paths to establish feature pyramids with multi-scale image features. We propose the Text-guided CSPLayer (T-CSPLayer) and Image-Pooling Attention (I-Pooling Attention) to further enhance the interaction between image features and text features. The T-CSPLayer extends the cross-stage partial layers by incorporating text guidance into multi-scale image features, adopting the max-sigmoid attention after the last dark bottleneck block to aggregate text features into image features. The I-Pooling Attention aggregates image features to update text embeddings through max pooling on multi-scale features to obtain 3x3 regions, resulting in 27 patch tokens, then updates text embeddings using multi-head attention. During inference, the offline vocabulary embeddings can be re-parameterized into weights of convolutional or linear layers for deployment.

Region-Text Contrastive Loss. Given the mosaic sample and texts, YOLO-World outputs K object predictions. We leverage task-aligned label assignment to match predictions with ground-truth annotations and assign each positive prediction with a text index as the classification label. We construct the region-text contrastive loss through cross entropy between object-text similarity and object-text assignments. In addition, we adopt IoU loss and distributed focal loss for bounding box regression.

Pseudo Labeling with Image-Text Data. Rather than directly using image-text pairs, we propose an automatic labeling approach: (1) extract noun phrases using n-gram algorithm; (2) pseudo labeling using a pre-trained open-vocabulary detector (GLIP) to generate pseudo boxes; (3) filtering by employing pre-trained CLIP to evaluate relevance and filter low-relevance annotations. Through this approach, we sample and label 246k images from CC3M with 821k pseudo annotations.

YOLO-World is developed based on the MMYOLO and MMDetection toolboxes with three variants: small (S), medium (M), and large (L). For pre-training, the model is trained for 100 epochs on 32 NVIDIA V100 GPUs with a total batch size of 512, using AdamW optimizer with initial learning rate of 0.002 and weight decay of 0.05. The pre-training data includes Objects365 (V1, 365 categories, 609k images), GQA (621k images), Flickr30k (149k images), and CC3M (246k images with 821k pseudo annotations). The text encoder is frozen during pre-training. On the challenging LVIS dataset (1203 categories) in zero-shot evaluation, YOLO-World achieves 35.4 AP with 52.0 FPS on V100, outperforming GLIP, GLIPv2, and Grounding DINO with fewer model parameters. Compared to DetCLIP, YOLO-World achieves comparable performance (35.4 vs 34.4 AP) while obtaining a 20x increase in inference speed.

Pre-training Data. Compared to the baseline trained on Objects365, adding GQA significantly improves performance with an 8.4 AP gain on LVIS. This improvement is attributed to the richer textual information provided by the GQA dataset. Adding part of CC3M (8% of full datasets) further brings 0.5 AP gain with 1.3 AP on rare objects.

Ablations on RepVL-PAN. The proposed RepVL-PAN improves the baseline (YOLOv8-PAN) by 1.1 AP on LVIS, and the improvements are remarkable in terms of the rare categories (AP_r). The improvements become more significant when pre-trained with the GQA dataset.

Text Encoders. The CLIP text encoder obtains superior results over BERT (+10.1 AP for rare categories). Fine-tuning BERT brings improvements (+3.7 AP) while fine-tuning CLIP leads to severe performance drop, attributed to degradation of generalization when fine-tuning on O365's limited 365 categories.

Fine-tuning on COCO. For fine-tuning on COCO, the RepVL-PAN is removed for further acceleration given the small vocabulary size. YOLO-World achieves decent zero-shot performance on COCO and higher performance after fine-tuning compared to methods trained from scratch.

Fine-tuning on LVIS. Compared to oracle YOLOv8-L trained on full LVIS, YOLO-World-L outperforms by 7.2 AP and 10.2 AP_r, demonstrating the effectiveness of the pre-training strategy for large-vocabulary detection. YOLO-World, as an efficient one-stage detector, outperforms previous state-of-the-art two-stage methods on overall performance.

Open-Vocabulary Instance Segmentation. The authors extend YOLO-World for open-vocabulary instance segmentation (OVIS), benchmarking under COCO-to-LVIS and LVIS-base-to-LVIS settings. Fine-tuning only the segmentation head retains zero-shot capabilities, while fine-tuning all modules may degrade open-vocabulary performance despite improving overall LVIS metrics.

We present YOLO-World, a cutting-edge real-time open-vocabulary detector aiming to improve efficiency and open-vocabulary capability in real-world applications. We have reshaped the prevalent YOLOs as a vision-language YOLO architecture for open-vocabulary pre-training and detection and proposed RepVL-PAN, which connects vision and language information with the network and can be re-parameterized for efficient deployment. We further present the effective pre-training schemes with detection, grounding and image-text data to endow YOLO-World with a strong capability for open-vocabulary detection. Experiments demonstrate the superiority of YOLO-World in terms of speed and open-vocabulary performance and indicate the effectiveness of vision-language pre-training on small models, which is insightful for future research. We hope YOLO-World can serve as a new benchmark for addressing real-world open-vocabulary detection.