Compared to the rapid development of large-scale vision transformers (ViTs) in recent years, large-scale models based on CNNs have not been thoroughly explored. This work presents InternImage, a new large-scale CNN-based foundation model that can obtain comparable or even better performance than current state-of-the-art ViTs. Different from recent CNNs that adopt large dense kernels, InternImage takes deformable convolution as the core operator, so that the model not only has the large effective receptive field required for downstream tasks such as detection and segmentation, but also has adaptive spatial aggregation conditioned by input and task information.

Starting from InternImage, the authors explore how to design and train CNN-based foundation models at a large scale. By customizing a series of block-level, stage-level, and model-level designs, together with a tailored large-scale training strategy, InternImage is successfully scaled to over 1 billion parameters and trained on 427 million images. It achieves 65.4 mAP on COCO test-dev and 62.9 mIoU on ADE20K, outperforming current leading CNNs and ViTs on these benchmarks. The code and models are available at github.com/OpenGVLab/InternImage.

Since the introduction of AlexNet, convolutional neural networks have been the de facto standard for visual recognition. However, the emergence of Vision Transformers (ViTs) has shifted the paradigm significantly. Models like ViT, Swin Transformer, and their scaled variants have demonstrated remarkable performance on a wide range of vision benchmarks. A natural question arises: Can CNN-based foundation models also achieve comparable or even better performance than ViTs when equipped with similar operator-level and architecture-level designs?

To answer this question, the authors first identify two key differences between the core operators of modern CNNs and ViTs. First, multi-head self-attention (MHSA) provides long-range dependencies and adaptive spatial aggregation, whereas standard convolutions have limited and static receptive fields. Second, ViTs benefit from advanced architectural components such as Layer Normalization, Feed-Forward Networks (FFN), and GELU activation, which are absent in traditional CNN designs. Previous attempts to bridge this gap using very large kernels (e.g., 31x31) still lag behind state-of-the-art ViTs and introduce optimization difficulties.

In this work, the authors propose a different approach. Instead of using large dense kernels, they adopt deformable convolution v3 (DCNv3) as the core operator, which uses 3x3 convolution kernels with learnable sampling offsets. This design provides three key advantages: (1) flexible receptive fields that can be either short-range or long-range depending on the learned offsets; (2) input-adaptive spatial aggregation similar to MHSA; and (3) avoidance of the optimization problems inherent in dense large kernels. Combined with transformer-inspired architectural components and customized stacking and scaling rules, InternImage becomes the first CNN-based model effectively scaled to over 1 billion parameters while matching ViT-level performance.

Vision foundation models have evolved from AlexNet through a long line of CNN architectures including VGGNet, GoogLeNet, ResNet, DenseNet, and EfficientNet. These models progressively improved accuracy and efficiency through innovations in depth, width, connectivity patterns, and neural architecture search. More recently, Vision Transformers (ViT) and their hierarchical variants such as Swin Transformer, PVT, and CSWin have introduced self-attention mechanisms that provide global receptive fields and dynamic weight computation. This has sparked renewed interest in modernizing CNN architectures by borrowing techniques from ViTs, as seen in works like ConvNeXt that systematically incorporate transformer-inspired design choices into pure convolutional networks.

The success of large-scale pre-training in NLP has inspired similar efforts in computer vision. Zhai et al. extended ViT to 2 billion parameters, and Liu et al. developed 3 billion parameter Swin variants, demonstrating that model scaling is a crucial factor for achieving state-of-the-art performance. However, CNN-based large-scale models have significantly lagged behind transformer variants in both parameter count and downstream performance. Recent attempts to improve CNNs through large kernel designs — such as RepLKNet with 31x31 kernels and SLaK with up to 51x51 kernels — have narrowed the gap but still fall short of the best ViTs and require specialized re-parameterization techniques to stabilize training.

Deformable convolution was first introduced by Dai et al. as DCNv1, which augments the standard convolution grid with 2D learnable offsets, allowing each sampling point to shift from its regular position. DCNv2 further introduced modulation scalars that re-weight the contribution of each sampling point. While these operators have proven effective as plug-in modules in detection and segmentation backbones, they have not been explored as the core building block for designing large-scale foundation models. This paper bridges this gap by extending DCNv2 into DCNv3 with modifications tailored for foundation model design.

The authors begin by analyzing the limitations of standard convolution compared to multi-head self-attention (MHSA). Standard convolution suffers from two critical shortcomings: (1) the effective receptive field remains relatively small despite increasing network depth, constrained by the fixed kernel size (typically 3x3 or 5x5); and (2) static convolution weights impose strong inductive biases — specifically 2D locality and translation equivalence — which restrict the model's ability to learn from web-scale diverse data. In contrast, MHSA computes dynamic attention weights conditioned on input features, enabling both long-range dependencies and adaptive spatial aggregation.

Recall that DCNv2 computes its output as: y(p₀) = Σ_k=1^K w_k m_k x(p₀ + p_k + Δp_k), where p_k denotes regular grid sampling locations, Δp_k are learnable offsets, and m_k are modulation scalars. DCNv2 already shares favorable properties with MHSA: the learnable offsets enable variable-length receptive fields, and the input-conditioned modulation provides adaptive spatial aggregation. However, DCNv2 has several design choices that hinder its scalability to large-scale models: location-specific convolution weights, single-group operation, and element-wise sigmoid normalization of modulation scalars.

DCNv3 introduces three critical modifications. First, weight sharing among convolutional neurons: the original DCNv2 maintains location-specific weights for each sampling point, making parameter count grow linearly with the number of sampling points. DCNv3 decomposes this into depth-wise components (handling spatial modulation) and point-wise shared projections, dramatically reducing parameters. Second, the multi-group mechanism: inspired by group convolution and multi-head attention, DCNv3 splits spatial aggregation into G groups, each with individual sampling offsets Δp_mk and modulation scalars m_mk, enabling different groups to learn diverse spatial aggregation patterns from different representation subspaces. Third, softmax normalization: replacing the original element-wise sigmoid (which produces unstable output range [0, K]) with softmax normalization along the K sampling points, constraining the sum to 1 and stabilizing gradient flow in large-scale training.

The resulting DCNv3 operator is formulated as: y(p₀) = Σ_g=1^G Σ_k=1^K w_g m_gk x_g(p₀ + p_k + Δp_gk), where G denotes the number of aggregation groups, K is the number of sampling points per group, w_g ∈ R^C×C' is the group-wise projection weight with C' = C/G, and the modulation scalars m_gk are normalized via softmax. Compared to MHSA, DCNv3 is more efficient because it uses sparse sampling (K=9 for 3x3 kernel) rather than attending to all spatial locations, while still achieving adaptive spatial aggregation through the learned offsets and modulation.

The basic block of InternImage departs from the traditional bottleneck design used in ResNet and instead adopts a transformer-like structure. Each block consists of DCNv3 as the core operator, preceded by Layer Normalization (LN) and followed by a Feed-Forward Network (FFN) with GELU activation. The sampling offsets and modulation scales are predicted via separable convolution from input features. This design combines the spatial modeling strength of deformable convolution with the training stability and representation capacity of transformer-style components.

The stem layer consists of two 3x3 convolutions with stride 2 and padding 1, flanked by Layer Normalization and GELU activations, which reduce the input resolution by 4x before the first stage. Between consecutive stages, downsampling layers employ a single 3x3 convolution with stride 2 and padding 1 followed by Layer Normalization, providing a 2x spatial reduction. The overall architecture follows the four-stage hierarchical design common to both modern CNNs and ViTs, producing feature maps at 1/4, 1/8, 1/16, and 1/32 of the input resolution, which are compatible with standard downstream task heads such as FPN and UperNet.

To reduce the vast hyperparameter space when designing model variants, the authors propose four stacking rules that constrain the architecture. Rule 1: the channel dimensions of stages 2, 3, and 4 are determined by the first stage channel C₁ through fixed scaling factors (2x, 4x, 8x). Rule 2: group numbers in each stage correspond to their channel configurations. Rules 3 and 4: the block count follows an "AABA" pattern — stages 1, 2, and 4 have equal numbers of blocks (L₁), while stage 3 has significantly more blocks (L₃), concentrating the majority of computational capacity in the third stage. These rules reduce the full 12-dimensional hyperparameter space to just 4 variables: (C₁, C', L₁, L₃).

With the constrained search space, the authors discretize the remaining hyperparameters: C₁ ∈ {48, 64, 80}, L₁ ∈ {1, 2, 3, 4, 5}, and C' ∈ {16, 32}, producing 30 candidate models that are evaluated on ImageNet with a short training schedule. The optimal configuration (C₁=64, C'=16, L₁=4, L₃=18) yields approximately 30 million parameters and serves as the "origin model" (InternImage-T) from which all other variants are derived through systematic scaling.

Starting from the origin model, larger variants are obtained by jointly scaling depth and width using the compound scaling approach. The depth is scaled as D' = α^φ · D (where D = 3L₁ + L₃) and the width as C₁' = β^φ · C₁, subject to the constraint α · β^1.99 ≈ 2, ensuring that each scaling step approximately doubles the model's FLOPs. Through empirical evaluation of multiple (α, β) combinations, the authors select α = 1.09 and β = 1.36, which balances depth and width growth for optimal performance.

This scaling strategy produces six model variants spanning a wide range of computational budgets: InternImage-T (30M params), InternImage-S (50M), InternImage-B (97M), InternImage-L (223M), InternImage-XL (335M), and InternImage-H (1.08B params). The block configurations are: T has {4,4,18,4}, S has {4,4,21,4}, B has {4,4,21,4} with wider channels, L uses {5,5,22,5}, XL uses {5,5,24,5}, and the largest H variant uses {6,6,32,6} blocks across the four stages, with C₁ = 320 and C' = 32.

Image Classification on ImageNet. InternImage-T/S/B are trained for 300 epochs on ImageNet-1K, while InternImage-L/XL are first pre-trained on ImageNet-22K for 90 epochs then fine-tuned on ImageNet-1K for 20 epochs. For the largest model, InternImage-H is pre-trained on a joint dataset of 427 million images (Laion-400M, YFCC-15M, CC12M) for 30 epochs using the M3I pre-training approach, then fine-tuned on ImageNet-1K. Among small models, InternImage-T achieves 83.5% top-1 accuracy, surpassing ConvNeXt-T (82.1%) by 1.4 points. InternImage-B reaches 84.9%, outperforming ConvNeXt-B (83.8%) by 1.1 points. At the largest scale, InternImage-H achieves 89.6% on 640x640 inputs, approaching ViT-G/14's 90.5% with a gap of only 0.9 points.

Object Detection on COCO. On the standard Mask R-CNN 1x schedule, InternImage-T achieves 47.2 box AP, outperforming Swin-T (42.7) by 4.5 points and ConvNeXt-T (44.2) by 3.0 points. InternImage-B reaches 48.8 box AP, 1.8 points ahead of ConvNeXt-B (47.0). With the advanced DINO detector and Objects365 pre-training, InternImage-H achieves a record 65.4 mAP on COCO test-dev, surpassing the previous best FD-SwinV2-G (64.2) by 1.2 points while using 27% fewer parameters (2.18B vs 3.00B). Notably, InternImage-H achieves this without knowledge distillation, which was employed by competing methods, further highlighting the inherent strength of the DCNv3-based backbone for dense prediction tasks.

Semantic Segmentation on ADE20K. Using the UperNet framework with single-scale testing, InternImage-T achieves 47.9 mIoU, surpassing ConvNeXt-T (46.0) by 1.9 points. InternImage-B reaches 50.8 mIoU, ahead of RepLKNet-31B (49.9) by 0.9 points. At the large scale with Mask2Former and 896x896 crop inputs, InternImage-H achieves 62.9 mIoU, surpassing SwinV2-G (59.9) by 3.0 points with approximately 3x fewer parameters, and even slightly exceeding BEiT-3 (62.8 mIoU). On multi-scale evaluation, InternImage-H reaches 60.3 mIoU with UperNet, which alone surpasses SwinV2-G's 59.9 mIoU.

Effect of weight sharing. Comparing shared versus unshared convolution weights in DCNv3, weight sharing reduces parameters by 42.0% and GPU memory by 84.2% for InternImage-H, while maintaining nearly identical accuracy: 83.5% vs 83.6% on ImageNet and 47.2 vs 47.4 box AP on COCO. This demonstrates that weight sharing is essential for scaling DCN-based models to billions of parameters without sacrificing performance — the memory savings alone make large-scale training feasible.

Multi-group mechanism and softmax normalization. Removing the multi-group design (using a single group) leads to a 1.2-point drop on ImageNet (from 83.5% to 82.3%) and a significant 3.4-point drop on COCO (from 47.2 to 43.8 box AP). Visualization of learned sampling locations shows that different groups concentrate their sampling points in different spatial regions, confirming that the multi-group mechanism enables learning diverse spatial patterns. For the normalization choice, replacing softmax with element-wise sigmoid causes severe training instability, with ImageNet accuracy plummeting to 65.7% and COCO box AP falling to 38.7. This validates that softmax normalization is critical for stable training of DCN-based large-scale models.

Effective receptive field analysis. Using gradient-based visualization, the authors compare the effective receptive fields (ERFs) of ResNet-101 and InternImage-S. Before training, both models exhibit localized ERFs. After training, InternImage achieves near-global ERFs in stages 3 and 4, whereas ResNet's ERF remains relatively small even in its deepest layers. Interestingly, InternImage's ERF pattern differs from ViTs: while ViTs maintain global ERFs throughout all layers, InternImage shows a progressive expansion of the effective receptive field with increasing depth, suggesting it learns a hierarchical spatial representation — local details in early layers and global context in later layers.

This paper presents InternImage, which demonstrates that CNN-based foundation models can match or even surpass transformer-based models when properly designed with deformable convolution operators, transformer-inspired architectural components, and systematic scaling strategies. Through the proposed DCNv3 operator — featuring weight sharing, multi-group aggregation, and softmax normalization — the model effectively addresses the long-standing limitations of CNNs in receptive field flexibility and adaptive spatial aggregation. The comprehensive experimental results across image classification, object detection, and semantic segmentation confirm that InternImage achieves state-of-the-art results on COCO (65.4 mAP) and ADE20K (62.9 mIoU) while remaining parameter-efficient.

The authors also acknowledge limitations of the current work. The latency of DCN operators remains problematic for real-time downstream applications, as the irregular memory access patterns of deformable convolution are less hardware-friendly than standard convolutions or self-attention. Additionally, large-scale CNN research is still in its early stages, and InternImage's performance gap with methods that leverage even larger private datasets (such as JFT-3B) suggests room for further improvement. The authors position this work as a starting point for efficient large-scale CNN development, encouraging the community to continue exploring the potential of CNN-based vision foundation models.