We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision tasks. While existing large vision models excel in transfer learning, they struggle to perform a diversity of tasks with simple instructions, a capability that implies handling the spatial hierarchy and semantic granularity required by different tasks. We adopt a sequence-to-sequence architecture that integrates all tasks under a common language modeling objective, enabling the model to handle diverse vision tasks through text-based prompt instructions without task-specific architectural modifications.

To achieve this, we propose to use FLD-5B, a dataset of 5.4 billion comprehensive visual annotations on 126 million images, using an iterative strategy of automated annotation and model refinement. We adopt a multitask learning approach with the FLD-5B dataset, incorporating a diverse range of annotation types to train the model to handle spatial hierarchy and semantic granularity. Extensive evaluations on numerous tasks demonstrated Florence-2 to be a strong vision foundation model contender with unprecedented zero-shot and fine-tuning capabilities across tasks including captioning, object detection, visual grounding, and referring expression segmentation.

Universal representation for diverse vision-related tasks presents unique challenges, notably the need for comprehensive perceptual abilities. In natural language processing, task-agnostic foundation models such as GPT and BERT have achieved remarkable success by handling diverse tasks through simple text instructions. However, the vision domain has yet to achieve the same level of task universality. Vision tasks span a broad spectrum from image-level understanding (classification, captioning) to region-level localization (detection, grounding) and pixel-level prediction (segmentation), each demanding distinct output formats and spatial reasoning capabilities.

Two critical challenges hinder the development of a universal vision foundation model. First, existing vision datasets provide specialized annotations but lack comprehensiveness for unified learning — ImageNet focuses on classification labels, COCO provides detection boxes, and Flickr30k offers image-text pairs, yet no single dataset covers the full spectrum of visual annotations needed. Second, there is an absence of a unified pre-training framework that can seamlessly integrate spatial hierarchy (from image-level to pixel-level) and semantic granularity (from coarse categories to fine-grained descriptions) without requiring task-specific architectural modifications.

To address these challenges, we present Florence-2, which employs a sequence-to-sequence learning paradigm, integrating all tasks under a common language modeling objective. The model takes text-based prompts as task instructions and generates text-based outputs including region coordinates and segmentation polygons expressed as location tokens. To provide the comprehensive training data required, we construct FLD-5B through an automated data engine that combines specialist models for initial annotation, data filtering with quality heuristics, and iterative model-based refinement. This data-centric approach enables training a compact yet versatile foundation model that achieves strong performance across the full spectrum of vision tasks.

We evaluate three dominant pre-training paradigms for vision models. Supervised pre-training on ImageNet classification excels at recognition but lacks adaptability to diverse downstream tasks. Self-supervised methods such as SimCLR and MAE reveal intricate visual features but may overemphasize certain attributes while neglecting spatial reasoning. Weakly supervised approaches like CLIP and SAM yield only image-level understanding or single-task specialization, failing to capture the full range of visual perception required for universal tasks.

We formulate three interrelated learning objectives addressing different granularities of visual understanding. The first objective targets image-level understanding, training the model on classification and captioning tasks to develop holistic semantic comprehension. The second objective focuses on region/pixel-level recognition, incorporating detection, grounding, and segmentation tasks to build spatial localization capabilities. The third objective addresses fine-grained visual-semantic alignment, training on phrase-to-region correspondence to enable precise mapping between language and visual regions. By combining these three learning objectives in a multitask framework, our foundation model learns to handle different levels of detail and semantic understanding.

Florence-2 employs a sequence-to-sequence architecture consisting of two main components. The vision encoder uses DaViT (Dual Attention Vision Transformer) to convert input images into flattened visual token embeddings, maintaining spatial information through transformer-based encoding. The multi-modality encoder-decoder is a standard transformer that processes concatenated visual and prompt embeddings: the visual tokens V' and text prompt embeddings T_prompt are merged to form multi-modality input X = [V', T_prompt], which is then processed through transformer layers with cross-attention mechanisms to generate the output sequence.

A crucial innovation is the unified task formulation: all 13 tasks are reformulated as sequence-to-sequence translation problems. For spatial tasks, the tokenizer vocabulary is extended to include location tokens representing quantized coordinates: bounding boxes use the format (x_0, y_0, x_1, y_1), text detection uses quadrilateral representations, and segmentation employs polygon representations. The optimization uses a standard cross-entropy loss applied uniformly: L = -sum_i log P(y_i | y_{<i}, x), eliminating the need for task-specific prediction heads and enabling a truly unified training pipeline.

The construction of FLD-5B begins with curating 126 million images from five sources: ImageNet-22k, Object 365, Open Images, Conceptual Captions, and LAION, targeting diverse visual content. The annotation process follows a three-phase pipeline. In the initial annotation phase, specialist models (both offline and cloud-based) generate synthetic labels across annotation types, which are merged with pre-existing human annotations where available. The data filtering phase applies text parsing using SpaCy to extract semantic elements, removes images with excessive objects, filters bounding boxes by confidence thresholds, and applies non-maximum suppression to eliminate redundant detections.

The third phase employs iterative refinement, where model predictions are used to progressively improve annotation quality. In each refinement cycle, the current model generates predictions on the training set; these refined predictions are merged with original annotations and used for the next training iteration. For tasks with sparse initial annotations, the pipeline employs task-specific fine-tuning followed by specialist re-annotation to bootstrap coverage. This self-improving loop is inspired by data flywheel concepts, where better models produce better annotations, which in turn train even better models — enabling annotation quality to scale beyond the capabilities of any individual specialist model.

The resulting FLD-5B dataset comprises 5.4 billion annotations across 126 million images, organized into three annotation categories. Text annotations include 500 million entries spanning brief descriptions (~8 tokens), detailed descriptions (~32 tokens), and more detailed descriptions (~71 tokens), providing multi-granularity textual coverage. Region-text pairs total 1.3 billion annotations with an average of 5.42 regions per image, linking spatial regions to semantic descriptions. Text-phrase-region triplets contribute 3.6 billion annotations, enabling fine-grained phrase grounding — the mapping from individual phrases within a caption to their corresponding visual regions.

Analysis of the dataset reveals several noteworthy characteristics. The description length distribution shows a long tail, with more detailed annotations providing substantially richer semantic content. The region annotations achieve broad coverage across object categories, with the iterative refinement process improving both spatial accuracy and semantic consistency compared to initial specialist predictions. Compared to existing datasets, FLD-5B offers orders of magnitude more annotations than COCO (2.5M annotations) or Visual Genome (108M region descriptions), while maintaining competitive annotation quality as validated through human evaluation on sampled subsets.

Florence-2 is evaluated in three settings: zero-shot evaluation across tasks, generalist model performance with public supervised data, and downstream task fine-tuning. In zero-shot evaluation, the model demonstrates remarkable capabilities without any task-specific training data. On COCO captioning, Florence-2-L achieves 135.6 CIDEr, substantially outperforming Flamingo (84.3 CIDEr) despite the latter having 80 billion parameters. On Flickr30k grounding, the model achieves 84.4 Recall@1, a 5.7-point improvement over Kosmos-2. For RefCOCO referring expression comprehension, Florence-2 demonstrates 4-8% absolute improvements over prior models, and uniquely provides referring expression segmentation capability (35.8% mIOU) that was absent in previous foundation models.

When fine-tuned as a generalist model with public supervised data, Florence-2 achieves 143.3 CIDEr on COCO captioning, competitive with models having 7.8 to 80 billion parameters. On TextVQA, the model reaches 73.5% accuracy without external OCR modules, and on RefCOCO referring expression comprehension, it achieves 95.3% accuracy, surpassing PolyFormer by 3.0 points. These results are achieved with only 0.77 billion parameters, demonstrating the effectiveness of comprehensive multi-task pre-training in producing compact yet powerful models.

As a pre-trained backbone for downstream tasks, Florence-2 shows substantial improvements over standard baselines. Using Mask R-CNN for COCO object detection, the Florence-2 backbone achieves 53.6 AP_box with 1x training schedule, compared to 46.7 AP for ImageNet-1k supervised initialization — a gain of 6.9 AP. With the DINO detector, the model reaches 59.2 AP, a 4.2-point improvement over ViT-B. For ADE20K semantic segmentation, Florence-2 achieves 54.9 mIOU, a 4.9-point improvement with 4x training efficiency compared to ImageNet pre-training. Notably, even with a frozen backbone, the model achieves competitive results with only 1.6-2.4 point drops, demonstrating strong generic visual representations.

Ablation studies validate the key design decisions. For multitask transfer, models trained with all three learning objectives (image + region + pixel) consistently outperform variants trained on subsets. An image-only model performs well on classification but poorly on detection; adding region objectives improves detection at the cost of classification; only the full three-tier model excels across all tasks simultaneously. Model scaling experiments show that Florence-2-L significantly outperforms Florence-2-B across all benchmarks, indicating capacity for further scaling. Data scaling analysis reveals consistent performance improvements as dataset size increases from 0.12M to 12M images, though with diminishing returns at larger scales, suggesting an eventual saturation point.

Vision-language foundation models can be broadly categorized into contrastive approaches and autoregressive approaches. Contrastive methods such as CLIP and Florence-1 align visual and textual representations through contrastive learning, excelling at zero-shot classification and retrieval but lacking generative capabilities for tasks like captioning or spatial prediction. Autoregressive methods including GIT, CoCa, and Flamingo generate textual outputs conditioned on visual inputs. Flamingo relies on frozen vision encoders and a Perceiver Resampler, limiting its ability to learn task-specific visual features. Florence-2 distinguishes itself through a fully trainable unified seq2seq architecture without frozen components, enabling end-to-end optimization across all modalities and tasks.

On the dataset front, prior work ranges from curated academic datasets like COCO (330K images, 2.5M annotations) and Visual Genome (108K images, 108M region descriptions) to web-scale datasets such as LAION (5B image-text pairs) and WIT (400M pairs). Academic datasets offer high-quality annotations but limited scale; web-scale datasets provide vast quantities but only image-level text associations without spatial annotations. FLD-5B bridges this gap by combining large-scale autonomous annotations with iterative refinement, exceeding prior work in both annotation density (5.4B across 126M images) and diversity (text + region + triplet annotations).

We have presented Florence-2, a unified vision foundation model that handles a diverse range of computer vision tasks through prompt-based instructions and a sequence-to-sequence architecture. The model is enabled by FLD-5B, a comprehensive dataset of 126 million images paired with 5.4 billion comprehensive annotations constructed via an automated data engine with iterative refinement. Our multitask learning framework addresses both spatial hierarchy and semantic granularity, training the model to excel across image-level, region-level, and pixel-level tasks simultaneously.

Florence-2 has exhibited remarkable zero-shot capabilities that extend across a wide spectrum of visual tasks, such as captioning, object detection, visual grounding, and referring expression segmentation. The universal representation pre-trained by Florence-2 substantially enhances downstream tasks across detection, segmentation, and vision-language alignment through efficient, unified training. These results establish that comprehensive multitask pre-training on diverse, high-quality annotations is a viable and effective path toward universal vision foundation models, analogous to the progress achieved by large language models in NLP.