We present ImageBind, an approach to learn a joint embedding across six different modalities — images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models.

Humans understand the world by absorbing multiple senses simultaneously — sight, sound, touch, smell, and movement. This holistic understanding enables us to recognize objects not just visually, but through their associated sounds, textures, and spatial configurations. Multimodal learning in AI aims to replicate this capability by learning representations that capture the relationships between different sensory modalities. A key challenge is that obtaining large-scale paired data across all possible modality combinations is impractical — while image-text pairs are abundant on the web, acquiring aligned data for combinations like audio-depth or thermal-IMU is prohibitively expensive.

Recent vision-language models such as CLIP and ALIGN have demonstrated remarkable success in learning aligned image-text representations from web-scale data. These models enable powerful zero-shot transfer capabilities for visual recognition tasks. However, they are fundamentally limited to two modalities — images and text. Extending such approaches to additional modalities like audio, depth, thermal, or inertial data would naively require paired datasets for every combination of modalities, resulting in an O(n^2) scaling problem. Prior multimodal approaches have attempted to handle three or more modalities, but typically require explicit supervision across all modality pairs or resort to limited modality combinations.

We introduce ImageBind, which learns a single joint embedding space across six modalities by using images as a binding modality. Our key insight is that images co-occur naturally with a wide variety of other modalities — videos contain synchronized audio, RGB-D cameras capture aligned depth maps, thermal cameras provide paired thermal imagery, and wearable cameras record simultaneous IMU data. By aligning each modality's embedding to the image embedding using contrastive learning on naturally paired data, we achieve an emergent alignment across all modalities without requiring any explicit pairing between non-image modalities. This approach scales linearly O(n) with the number of modalities, rather than quadratically.

Vision-language pretraining has made significant strides with models like CLIP, ALIGN, and Florence that learn aligned image-text representations from large-scale web-crawled datasets containing hundreds of millions to billions of image-text pairs. These models use contrastive learning objectives to pull together matched image-text pairs while pushing apart unmatched ones in a shared embedding space. The resulting representations exhibit strong zero-shot transfer capabilities, enabling classification, retrieval, and detection without task-specific training. However, these approaches are inherently bimodal and cannot directly accommodate modalities beyond images and text.

Multimodal learning beyond vision and language has explored various combinations of sensory data. AudioCLIP extends CLIP to incorporate audio, while VATT learns from video, audio, and text simultaneously. Other works focus on specific pairings such as image-depth or audio-visual correspondence. A common limitation is that these methods are designed for specific modality combinations and do not generalize to arbitrary modalities. Furthermore, approaches that do handle multiple modalities typically require paired data across all modality combinations they aim to support, which limits their scalability. Self-supervised learning methods have shown that effective representations can be learned from unpaired data within a single modality, but cross-modal alignment still generally requires paired supervision.

The idea of using a shared embedding space to connect different domains has a long history. Metric learning and contrastive losses have been widely adopted for learning semantically meaningful distance functions. The InfoNCE loss, used in contrastive learning frameworks, provides a principled approach to aligning representations by treating the task as an instance discrimination problem. Recent work has shown that the quality of the embedding space depends critically on the scale and diversity of training data, as well as the capacity of the encoder models. ImageBind builds upon these foundations but introduces the novel insight that a single binding modality (images) can serve as a universal connector, eliminating the need for explicit cross-modal pairings between all modalities.

ImageBind creates a joint embedding space by leveraging images as a binding modality that connects all six modalities. The approach is based on a simple observation: images naturally co-occur with diverse modalities. Videos provide naturally aligned (image, text), (image, audio) pairs; RGB-D sensors provide (image, depth) pairs; thermal cameras provide (image, thermal) pairs; and wearable devices with cameras provide (video, IMU) pairs. Rather than requiring paired data between all C(M, 2) = M(M-1)/2 combinations of M modalities, we only need M-1 pairs, each involving images. This reduces data requirements from quadratic to linear in the number of modalities.

For each modality pair (I, M) where I denotes images and M denotes another modality, we train modality-specific encoders using the InfoNCE contrastive loss. Given a batch of N aligned pairs {(I_i, M_i)}, the loss encourages the embeddings of matched pairs to be close while pushing apart unmatched pairs in the joint space. Specifically, we normalize the embeddings to lie on a unit hypersphere and use a symmetric cross-entropy loss over cosine similarities scaled by a learnable temperature parameter. The image encoder is initialized from a pretrained CLIP model and kept frozen, while the modality-specific encoders are trained from scratch or from pretrained checkpoints. This design ensures that all modalities are aligned to the same image embedding space, and hence to each other, through the transitive property of embedding alignment.

The choice of images as the binding modality is motivated by several factors. First, images are the most information-rich sensory modality — they capture spatial structure, object appearance, scene layout, and contextual cues that correlate with many other sensory experiences. Second, large-scale pretrained image encoders (such as those in CLIP) already encode rich semantic representations that have been shown to transfer well across tasks. Third, images naturally co-occur with the widest range of other modalities due to the ubiquity of cameras in modern sensor systems. By leveraging this natural centrality of images, ImageBind avoids the need for specialized data collection pipelines for non-image modality pairs.

Formally, let f_I denote the image encoder and f_M denote a modality-specific encoder. For a pair (I_i, M_i), we compute normalized embeddings q_i = f_I(I_i) / ||f_I(I_i)|| and k_i = f_M(M_i) / ||f_M(M_i)||. The InfoNCE loss for modality M is defined as L_M = -1/N sum_i log( exp(q_i . k_i / tau) / sum_j exp(q_i . k_j / tau) ), where tau is the learnable temperature. The total training objective is the sum of losses across all modality pairs: L = sum_{M in {T, A, D, Th, IMU}} L_M. This formulation ensures that each modality encoder learns to map its inputs into the same region of the embedding space as the corresponding image features.

The image encoder uses a Vision Transformer (ViT) architecture, specifically ViT-H/14 from OpenCLIP, which provides a strong pretrained representation. For text encoding, we use the corresponding CLIP text encoder. The audio encoder processes 2-second audio clips converted to mel-spectrograms and treated as 2D images, using a ViT architecture. The depth encoder processes depth maps similarly as single-channel images with a ViT. The thermal encoder handles thermal infrared images, also processed with a ViT. The IMU encoder uses a Transformer architecture that processes 5-second windows of accelerometer and gyroscope data projected to a common dimension. All modality encoders share the same embedding dimensionality to enable direct comparison in the joint space.

The training data leverages naturally paired datasets for each modality. For (image, text) pairs, we use web-crawled data following the CLIP paradigm. For (video, audio) pairs, we use the AudioSet dataset which contains YouTube video clips with audio annotations, treating video frames as images. For (image, depth) pairs, we use ScanNet, a large-scale indoor RGB-D dataset, and SUN RGB-D. For (image, thermal) pairs, we use the LLVIP dataset containing visible-infrared image pairs. For (video, IMU) pairs, we leverage the Ego4D dataset captured from egocentric wearable cameras equipped with IMU sensors. Importantly, these datasets are collected independently and no cross-modal pairing is needed between non-image modalities.

A critical design choice is that the image encoder remains frozen during training — only the modality-specific encoders are updated. This serves two purposes: first, it preserves the rich semantic structure already learned by the CLIP image encoder, preventing catastrophic forgetting; second, it ensures that the image embedding space remains stable as an anchor, so that all modalities trained at different times or on different datasets are aligned to the same reference frame. The modality encoders are trained using large batch sizes with distributed training across multiple GPUs. Data augmentation strategies are applied according to each modality's characteristics — spatial augmentations for images and depth, temporal augmentations for audio and IMU.

A remarkable property of ImageBind is its emergent zero-shot capabilities. Since all modalities are aligned to the same embedding space, and text embeddings from CLIP already support zero-shot classification for images, this capability automatically transfers to all other modalities. For instance, even though the audio encoder was never trained with paired (audio, text) data, it can perform zero-shot audio classification by comparing audio embeddings to text embeddings of class names. Similarly, depth maps, thermal images, and IMU signals can all be classified in a zero-shot manner using text prompts, despite never having seen text during their encoder training. This emergent transfer arises purely from the transitive alignment through the shared image embedding space.

The shared embedding space also enables modality arithmetic — adding or subtracting embedding vectors from different modalities to create composite queries. For example, combining an image embedding of a dog with an audio embedding of ocean waves can retrieve images of dogs at the beach. This arithmetic compositionality suggests that the embedding space captures meaningful semantic relationships that are consistent across modalities. Furthermore, ImageBind's embeddings can be used with existing generative models — by replacing CLIP image embeddings with audio or other modality embeddings in text-to-image generation pipelines like DALL-E 2, one can generate images from audio inputs (e.g., generating rainforest scenes from rainforest sounds) or from other non-visual modalities, without any modification to the generation model itself.

An important finding is that ImageBind's emergent capabilities scale with the strength of the image encoder. When upgrading from a smaller ViT-B to the larger ViT-H image encoder, emergent zero-shot performance across all non-image modalities improves substantially, even though only the image encoder changed and the modality-specific encoders remain the same size. This suggests that a stronger visual representation provides a richer anchor point for cross-modal alignment, and that improvements in vision models can automatically benefit understanding in other sensory modalities. This scaling property also positions ImageBind as a new evaluation framework for vision models — measuring their impact on non-visual downstream tasks.

We evaluate ImageBind's emergent zero-shot classification capabilities across multiple modalities and benchmarks. For audio classification, we evaluate on ESC-50 and AudioSet using text embeddings of category names. ImageBind achieves state-of-the-art zero-shot audio classification, outperforming AudioCLIP and other specialized audio-text models, despite never being trained on paired audio-text data. For depth classification, we evaluate on NYU-v2 and SUN RGB-D scene recognition tasks, where ImageBind achieves strong zero-shot performance. For thermal classification, we evaluate on LLVIP and related benchmarks. Across all modalities, ImageBind's zero-shot performance matches or exceeds that of models specifically supervised for those modality-text pairs.

Beyond zero-shot evaluation, we assess ImageBind's few-shot classification performance. Using a simple linear probe on frozen ImageBind features, we evaluate performance with 1, 2, and 4 labeled examples per class. On audio tasks, ImageBind achieves approximately 40 percentage points improvement in top-1 accuracy on 4-shot classification compared to Meta's AudioMAE models, both self-supervised and supervised variants. This dramatic improvement demonstrates that ImageBind's embeddings provide a semantically rich starting point that requires very few labeled examples to adapt to specific tasks. The few-shot results also show consistent improvements across depth, thermal, and IMU modalities, confirming that the joint embedding space captures transferable semantic structure across all modalities.

We conduct ablation studies to understand the contribution of key design choices. First, we examine the impact of the image encoder strength: upgrading from ViT-B to ViT-L to ViT-H consistently improves emergent zero-shot performance across all modalities, with particularly large gains on audio (+15.7% on ESC) and depth (+8.3% on NYU-v2). Second, we ablate the importance of the frozen image encoder versus fine-tuning it jointly — results show that fine-tuning the image encoder degrades cross-modal alignment, confirming that a stable anchor is essential. Third, we study the effect of training data scale: reducing the training data for any single modality pair primarily affects that modality's alignment quality but has minimal impact on other modalities, validating the independence of the modality-specific training.

We further evaluate cross-modal retrieval between non-image modality pairs. For audio-to-text retrieval, text-to-depth retrieval, and audio-to-depth retrieval, ImageBind demonstrates meaningful retrieval performance despite never having seen these modality combinations during training. The retrieval quality correlates with the semantic similarity between the modalities and images — modalities that have richer visual content (like depth) tend to show better cross-modal retrieval than more abstract modalities (like IMU). We also evaluate embedding space quality through nearest-neighbor analysis, showing that semantically similar concepts cluster together across modalities in the joint space. The audio of a guitar, the image of a guitar, and the text "guitar" all map to nearby regions in the embedding space, confirming the semantic coherence of the joint representation.

Finally, we demonstrate cross-modal generation by integrating ImageBind with DALL-E 2's image decoder. By feeding audio embeddings from ImageBind into the DALL-E 2 decoder (which was trained to generate images from CLIP embeddings), we achieve audio-to-image generation without any additional training. The generated images semantically correspond to the input audio — rainforest sounds produce lush jungle scenes, bustling market sounds produce crowded market images, and bird songs produce images of birds in natural settings. This plug-and-play compatibility with existing generative models demonstrates the practical utility of aligning all modalities to a well-established embedding space, and opens new avenues for multimodal content creation that was previously inaccessible.

We have presented ImageBind, a method for learning a joint embedding space across six modalities using only image-paired data. Our key insight is that images serve as a natural binding modality — their rich semantic content and ubiquitous co-occurrence with other modalities make them an ideal anchor for cross-modal alignment. Through the transitive property of contrastive embedding alignment, ImageBind achieves emergent capabilities such as zero-shot classification, cross-modal retrieval, modality arithmetic, and cross-modal generation for modality pairs that were never explicitly aligned during training. Our results demonstrate that this approach outperforms specialist supervised models on emergent zero-shot tasks, validating the effectiveness of the binding-through-images paradigm.

Several limitations and future directions remain. The quality of cross-modal alignment is inherently bounded by the quality of image-modality pairing — modalities with weaker or noisier visual correspondence (such as IMU) show lower alignment quality. The transitive alignment is approximate rather than exact, and the accumulated alignment error grows as more modalities are chained through the image anchor. Furthermore, the current approach does not model temporal dynamics for sequential modalities like audio and IMU, treating them as fixed-length windows. Future work could explore dynamic binding with temporal models, extending to additional modalities such as touch and smell, and investigating whether the binding paradigm can be applied to other anchor modalities beyond images.

More broadly, ImageBind represents a step toward holistic AI systems that can perceive and reason across multiple sensory modalities, mirroring the multimodal nature of human perception. The finding that stronger vision models improve non-visual tasks suggests a deep connection between visual understanding and general-purpose representation learning. As vision models continue to improve in scale and capability, ImageBind's binding paradigm provides a principled and scalable pathway for extending these advances to the full spectrum of sensory experience. We release our models and code to facilitate further research in multimodal representation learning.