The exponential growth of large language models (LLMs) has opened up numerous possibilities for multimodal AGI systems. However, in the field of computer vision, the weights of large-scale vision foundation models are often employed as the backbone of other models. The scaling up of vision foundation models and their alignment to LLMs has not been sufficiently explored. In this work, the authors present InternVL, scaling up the vision foundation model to 6 billion parameters and progressively aligning it with an LLM, using web-scale image-text data from diverse sources. The model is versatile, and can be applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks including visual perception tasks such as image-level or pixel-level recognition, vision-language tasks such as zero-shot image/video classification, zero-shot image-text retrieval, and link with LLMs to create multimodal dialogue systems. It has powerful visual capabilities and can be a good alternative to the ViT-22B.

Recent vision-language foundation models (VLLMs) typically connect a vision encoder to a large language model using lightweight "glue" layers. However, this paradigm creates three critical limitations: first, there is a disparity in parameter scale — vision encoders hover around 1 billion parameters while LLMs exceed 100 billion, creating an imbalanced multimodal system. Second, the vision and language components learn inconsistent representations because they are pre-trained on different objectives and datasets. Third, the connection between the two modalities is often inefficient, relying on simple projection layers that fail to bridge the representational gap.

To address these limitations, the authors propose InternVL with three key design principles. First, parameter-balanced components: the vision encoder (InternViT-6B) is scaled to 6 billion parameters, and the language middleware (QLLaMA) contains 8 billion parameters, creating a more balanced multimodal architecture. Second, consistent representation alignment: by initializing QLLaMA from a pre-trained multilingual LLaMA, the vision and language components share a common representational foundation. Third, a progressive image-text alignment strategy that advances from contrastive learning to generative learning, ensuring robust cross-modal alignment at each stage.

The contributions of this work are threefold. First, InternVL scales the vision foundation model to 6 billion parameters, making it the largest open-source vision encoder to date. Second, the progressive alignment strategy enables the model to seamlessly bridge contrastive and generative learning paradigms within a unified framework. Third, the model achieves state-of-the-art performance on 32 benchmarks spanning visual perception, vision-language understanding, and multimodal dialogue, demonstrating broad versatility without task-specific architectural modifications.

Vision foundation models have evolved from AlexNet through ResNet to the Vision Transformer (ViT) family. Recent scaling efforts include ViT-22B and EVA-02, which demonstrate that larger vision models yield better representations for downstream tasks. However, these models primarily derive their training from visual-only datasets or BERT-style alignment, lacking direct integration with LLMs. This disconnect means that even the most powerful vision encoders cannot natively participate in language-driven reasoning, limiting their utility in the era of multimodal AI.

Large language models such as the GPT series, LLaMA, and InternLM have demonstrated extraordinary capabilities in reasoning, instruction following, and in-context learning. The integration of visual perception into these models has spawned a rich landscape of vision-language large models (VLLMs), including Flamingo, BLIP-2, and LLaVA. These approaches typically employ a frozen or lightly fine-tuned vision encoder (often CLIP ViT-L) connected to an LLM decoder via a projection layer or Q-Former module. While effective, the vision encoder remains the bottleneck — its limited capacity constrains the visual information available to the language model, and its representations are not natively aligned with the LLM's embedding space.

Unlike traditional vision-only or dual-encoder approaches, InternVL combines two core components: InternViT-6B, a vision transformer with 6 billion parameters, and QLLaMA, a language middleware with 8 billion parameters initialized from multilingual LLaMA. The architecture supports flexible composition for diverse visual-linguistic tasks through three inference modes: (1) visual perception, where InternViT-6B generates feature maps for dense prediction tasks; (2) contrastive tasks, where attention pooling extracts global visual features for similarity computation; and (3) generative tasks, where QLLaMA reorganizes visual representations as prefix tokens for sequential generation.

The design of InternViT-6B involves systematic hyperparameter exploration, varying depth {32, 48, 64, 80}, head dimension {64, 128}, and MLP ratio {4, 8} across 16 model variants. Evaluation reveals that "depth, head dimension, and MLP ratio have little impact on the performance" for a fixed parameter count, suggesting that the total number of parameters matters more than the specific architectural configuration. The final selected architecture uses width 3200, depth 48, MLP dimension 12800, and 25 attention heads, yielding 5.9 billion parameters. This variant is chosen based on a composite criterion of throughput, accuracy, and training stability.

QLLaMA is built upon a pre-trained multilingual LLaMA augmented with 96 learnable queries and cross-attention layers totaling 1 billion additional parameters that are randomly initialized. This design offers three advantages over the widely-used Q-Former: (1) pre-trained weight initialization provides a strong starting point for representation alignment, unlike Q-Former which is trained from scratch; (2) QLLaMA is 42 times larger than Q-Former, providing substantially greater capacity for cross-modal understanding; (3) the architecture is applicable to both contrastive and generative tasks, whereas Q-Former is primarily designed for contrastive alignment.

The alignment proceeds through three progressive stages. Stage 1: Vision-Language Contrastive Training aligns InternViT-6B with multilingual LLaMA-7B on web-scale noisy image-text pairs. The training dataset comprises 6.03 billion image-text pairs (4.98 billion after cleaning) drawn from diverse sources including LAION-en, LAION-multi, LAION-COCO, COYO, Wukong, CC3M/12M, and SBU. The loss function follows CLIP's symmetric cross-entropy formulation. This stage establishes the foundation for contrastive task excellence and visual perception capabilities.

Stage 2: Vision-Language Generative Training builds upon Stage 1 by inheriting its weights and introducing newly added cross-attention components. The training uses filtered, high-quality data reduced to 1.03 billion pairs after rigorous quality filtering. The loss function combines three objectives: image-text contrastive loss (ITC), image-text matching loss (ITM), and image-grounded text generation loss (ITG). Critically, both InternViT-6B and the base QLLaMA parameters remain frozen during this stage, with only the newly added cross-attention layers being optimized. This design preserves the strong visual representations established in Stage 1 while learning the generative bridge.

Stage 3: Supervised Fine-tuning connects InternVL with downstream LLM decoders such as Vicuna and InternLM. The fine-tuning uses approximately 4 million instruction-tuning samples spanning captioning, visual question answering, OCR, visual grounding, and multi-turn dialogue. The framework offers flexibility: one can freeze the LLM decoder while training only the MLP projection layer, or employ QLLaMA as a more powerful intermediary. This stage transforms the aligned vision-language model into a practical multimodal dialogue system capable of instruction-following and open-ended visual reasoning.

On visual perception benchmarks, InternViT-6B demonstrates exceptional capabilities. For image classification on ImageNet-1K, linear probing achieves 88.2% top-1 accuracy, which represents "the currently best linear evaluation results without the JFT dataset." On semantic segmentation (ADE20K), the model consistently outperforms ViT-22B across all evaluation protocols: few-shot learning shows consistent advantages across varied training data proportions; linear probing achieves 47.2 mIoU, surpassing ViT-22B by 12.6 points; and full-parameter tuning reaches 58.9 mIoU, a gain of 3.6 points over ViT-22B. These results confirm that scaling a vision encoder with language-aligned pre-training produces superior visual representations even for pure vision tasks.

On vision-language benchmarks, InternVL demonstrates broad competence. For zero-shot image classification, InternVL-C achieves leading performance with "stronger robustness to distribution shift, manifesting in consistent accuracy across ImageNet variants." In zero-shot video classification, single-frame evaluation yields 76.1%, 75.5%, and 67.5% on Kinetics-400/600/700, surpassing EVA-02 by +6.3, +6.2, and +4.1 points respectively; with 8-frame uniform sampling, the model outperforms ViCLIP by more than 3.3 points. For image-text retrieval, InternVL shows strong multilingual performance across both English and Chinese datasets, with InternVL-G (incorporating generative training) further enhancing retrieval accuracy.

On multimodal dialogue benchmarks, InternVL-Chat combined with Vicuna-13B achieves 1317.2 on MME (spanning 14 sub-tasks) and 87.6 recall on POPE (hallucination evaluation), demonstrating "superior performance compared with previous methods, under the condition of fair trainable parameter counts." Ablation studies further validate the design choices: minimalist supervised fine-tuning with only MLP training shows InternVL outperforms vision encoder baselines, confirming the value of the scaled vision encoder. Moreover, "significant improvement when using QLLaMA as the glue layer" validates the claim that feature representation alignment through a language-informed middleware yields substantial gains over simple projection layers.

InternVL bridges the gap between vision foundation models and large language models by scaling the vision encoder to 6 billion parameters and implementing a progressive alignment strategy. The research demonstrates "proficiency in a wide range of generic visual-linguistic tasks" spanning classification, retrieval, captioning, visual question answering, and dialogue. The three-stage training pipeline — from contrastive pre-training on 4.98 billion pairs, through generative training on 1.03 billion quality-filtered pairs, to supervised fine-tuning on 4 million instruction samples — provides a principled framework for building versatile multimodal systems that treat vision as a first-class citizen alongside language.