Segment Anything Model (SAM) has emerged as a powerful tool for numerous vision applications. A key component that drives the impressive performance for zero-shot transfer and high versatility is a super large Transformer model trained on the extensive high-quality SA-1B dataset. While beneficial, the huge computation cost of SAM model has limited its applications to wider real-world applications. To address this limitation, we propose EfficientSAMs, light-weight SAM models that exhibits decent performance with largely reduced complexity. Our idea is based on leveraging masked image pretraining, SAMI, which learns to reconstruct features from SAM image encoder for effective visual representation learning. Further, we take SAMI-pretrained light-weight image encoders and mask decoder to build EfficientSAMs, and finetune the models on SA-1B for segment anything task. We perform evaluations on multiple vision tasks including image classification, object detection, instance segmentation, and semantic segmentation, and find that our proposed pretraining method, SAMI, consistently outperforms other masked image pretraining methods. On segment anything task such as zero-shot instance segmentation, our EfficientSAMs with SAMI-pretrained lightweight image encoders perform favorably with a significant gain (e.g., ~4 AP on COCO/LVIS) over other fast SAM models.

Segment Anything Model (SAM) has been very successful in the vision field, achieving state-of-the-art performance in a variety of image segmentation tasks. Trained on SA-1B, which contains more than 1B masks from 11M images, SAM demonstrates remarkable zero-shot transfer capabilities across tasks including zero-shot edge detection, zero-shot object proposal generation, and zero-shot instance segmentation. The foundation of SAM lies in a Vision Transformer (ViT) architecture that enables its high versatility and applicability to a wide range of real-world vision applications.

Despite the foregoing advantages, the model of SAM turns out to be a major efficiency bottleneck for practical deployment since the architecture of SAM, especially, the image encoder (e.g., ViT-H) is very expensive. The ViT-H encoder contains 632M parameters while the decoder uses only 3.87M parameters, creating high computation and memory costs that impede real-time applications. This significant imbalance between encoder and decoder complexity motivates the search for lightweight alternatives that preserve SAM's core capabilities.

In this paper, we propose using a well-pretrained lightweight ViT image encoder (e.g., ViT-Tiny/-Small) to reduce the complexity of SAM while maintaining decent performance. Our approach, SAMI (SAM-leveraged masked image pretraining), leverages masked image pretraining by training lightweight encoders to reconstruct features from SAM image encoder rather than reconstructing image patches. This knowledge-transfer mechanism produces generalized backbones applicable to diverse downstream tasks. The methodology involves two stages: pretraining SAMI on ImageNet-1K, then finetuning on SA-1B for segment anything tasks. The authors emphasize three key contributions: (1) proposing the SAMI framework for improved masked pretraining; (2) demonstrating that SAMI-pretrained backbones generalize across classification, detection, and segmentation; (3) delivering EfficientSAMs as complementary to SAM for practical deployment with state-of-the-art efficiency-quality tradeoffs.

Segment Anything Model (SAM) represents a milestone vision foundation model enabling object segmentation via interaction prompts. Its remarkable zero-shot transfer extends beyond segmentation to applications including in-painting, image restoration, image editing, image shadow removal, object tracking, and 3D object reconstruction. Recent works address practical deployment by proposing efficiency improvements: FastSAM develops a CNN-based architecture, YOLOv8-seg, while MobileSAM presents decoupled distillation for obtaining a lightweight image encoder. The present work aims at addressing this efficiency issue for practical deployment of SAM.

Knowledge distillation transfers dark knowledge from large teacher models to smaller students through soft labels from a teacher model. Advanced approaches include decoupled distillation separating target class and non-target class knowledge, and feature-based distillation from intermediate layers. In the domain of masked image pretraining, self-supervised approaches employ denoising autoencoders and context encoders for Vision Transformers. BEiT adopted masked image modeling (MIM) to predict visual tokens, while MAE and SimMIM achieve effective visual representation learning through direct pixel reconstruction. MAE-based extensions use large teacher models to guide pretraining, establishing the foundation for the authors' SAMI approach.

Vision Transformers demonstrate advantages of generalization over their CNN counterparts. Smaller variants like ViT-Small and ViT-Tiny complement larger models for efficiency-sensitive scenarios. MobileViT explores combining ViT with convolutions, achieving better generalization than lightweight CNNs with reduced memory and computation. Related architectures such as LeViT, EfficientFormer, Next-ViT, and Tiny-ViT represent orthogonal progress independent of the EfficientSAM work, as EfficientSAM's contribution focuses on pretraining strategy rather than architectural design.

Masked Autoencoders (MAE) model has two components, an encoder and a decoder, both using Transformer layers. MAE processes input images into non-overlapping patches, grouping them into unmasked tokens (processed by the encoder) and masked tokens (serving as reconstruction targets). The framework employs a high mask ratio (e.g., 75%) to prevent information leakage through neighbor extrapolation. Only the unmasked tokens are fed into the encoder, making pretraining efficient; the decoder then takes both encoded unmasked tokens and learnable mask tokens to reconstruct the original image patches at masked positions.

SAMI adapts the MAE framework to leverage SAM's image encoder as the reconstruction target. Rather than reconstructing pixels, the method trains lightweight encoders to align with latent features from SAM image encoder, thereby transferring knowledge embedded in SAM. The decoder architecture uses a cross-attention mechanism where queries come from masked tokens while keys and values are derived from both unmasked features from the encoder and masked features. Output features from masked and unmasked tokens are merged and reordered to their original image positions.

Features from the encoder and decoder pass through a linear projection head for aligning the features from SAM image encoder, addressing the feature dimension mismatch between the lightweight encoder and SAM's ViT-H. The reconstruction loss minimizes the mean squared error between SAM encoder features and SAMI's projected output: L = (1/N) * sum ||f_sam(x) - f_h(x)||^2, where N represents the token count, f_sam is the SAM encoder output, and f_h combines encoder-decoder outputs through linear projection. After SAMI pretraining, the decoder is discarded and the lightweight encoder (ViT-Tiny/-Small/-Base) serves as SAM's image encoder replacement.

After SAMI pretraining on ImageNet-1K, the pretrained lightweight ViT encoder is coupled with SAM's original mask decoder to form EfficientSAM. The complete model is then finetuned end-to-end on the SA-1B dataset for segment anything tasks. Two primary variants are proposed: EfficientSAM-Ti using ViT-Tiny (5M parameters) and EfficientSAM-S using ViT-Small (22M parameters). Compared to SAM's ViT-H with 632M parameters, EfficientSAM-S achieves approximately 20x reduction in parameters and inference time while targeting minimal performance degradation.

On ImageNet-1K image classification, SAMI demonstrates consistent improvements across all model scales. SAMI-B achieves 84.8% top-1 accuracy, outperforming MAE-B (83.6%), iBOT-B (84.4%), CAE-B (83.9%), and BEiT-B (83.7%) by margins of 1.2%, 0.4%, 0.9%, and 1.1% respectively. For lightweight models, the gains are even more pronounced: SAMI-S reaches 82.7% versus MAE-S (81.5%), DeiT-S (81.2%), and SSTA-S (81.4%). SAMI-Ti achieves 76.8% over MAE-Ti (75.2%), representing a 1.6% improvement. Notably, SAMI requires only 400 epochs of pretraining compared to 1600 epochs for standard MAE, demonstrating superior training efficiency.

For object detection and instance segmentation on COCO using the ViTDet framework, SAMI shows substantial improvements. SAMI-B achieves 52.5 AP_bbox and 46.5 AP_mask, gaining 0.9 AP_bbox and 0.6 AP_mask over MAE-B. More significantly, SAMI-Ti reaches 44.7 AP_bbox and 40.0 AP_mask versus MAE-Ti's 37.9/34.9, a remarkable 6.8 AP_bbox and 5.1 AP_mask improvement. On ADE20K semantic segmentation using Mask2former, SAMI-B achieves 51.8 mIoU (3.7 improvement over MAE-B's 49.3), SAMI-S reaches 48.8 mIoU (4.7 gain), and SAMI-Ti achieves 42.7 mIoU (3.7 gain). These results confirm that SAMI-pretrained backbones transfer effectively across dense prediction tasks.

On the segment anything task, EfficientSAM-S achieves 76.9 mIoU (1-box) and 50.0 mIoU (1-click) on COCO, and 75.4 mIoU (1-box) and 56.2 mIoU (1-click) on LVIS. EfficientSAM-Ti reaches 75.7 mIoU (1-box) and 45.5 mIoU (1-click) on COCO. For zero-shot instance segmentation, EfficientSAM-S achieves 44.4 AP on COCO and 42.3 AP on LVIS, gaining 6.5 and 7.8 AP respectively over FastSAM. Critically, EfficientSAM-S uses only 25M parameters versus FastSAM's 68M, while the gap with the full SAM (46.5 AP) is only 2.1 AP on COCO. This demonstrates that EfficientSAM provides an excellent efficiency-performance tradeoff for practical deployment.

Ablation studies validate key design choices. For reconstruction loss, MSE loss outperforms cosine similarity: SAMI-S achieves 82.7% (MSE) versus 82.3% (cosine) on ImageNet-1K. The cross-attention decoder with masked tokens only improves SAMI-Ti performance by approximately 3% over standard MAE's approach of processing all tokens. For mask ratio, optimal performance occurs at 75%: SAMI-B achieves 84.8% at 75%, versus 84.6% (50%) and 84.7% (85%). Regarding alternative reconstruction targets, using CLIP encoder features improves MAE-Ti by 0.8%, but SAM features produce stronger results due to SAM's richer spatial understanding. Finally, even at 0.1 finetuning epochs, EfficientSAMs achieve decent performance, reaching over 2.5 mIoU improvement at 1 epoch.

We proposed a masked image pretraining approach, SAMI, to explore the potential of ViTs under the guidance of SAM foundation model. By reconstructing latent features from SAM image encoder, the method transfers knowledge from vision foundation model to lightweight ViTs. Comprehensive experiments validate SAMI's advantages across image classification, object detection and instance segmentation, semantic segmentation, and the segment anything task. The resulting EfficientSAMs demonstrate that practical deployment of segment anything capabilities is achievable with significantly reduced computational cost. The work suggests potential applications beyond efficient segment anything, indicating that SAMI's pretraining paradigm may benefit other vision foundation model compression scenarios.