Images of organisms are increasingly available and becoming an important source of biological information for evolutionary biology, ecology, and biodiversity research. However, there is currently no general-purpose vision model applicable to the wide variety of organismal biology questions and image sources. The authors address this gap by curating and releasing TreeOfLife-10M, the largest and most diverse ML-ready dataset of biology images, containing over 10 million images representing 454,000 taxa. They develop BioCLIP, a foundation model that leverages multimodal contrastive learning combined with taxonomic structure. BioCLIP consistently and substantially outperforms existing baselines by 17% to 20% absolute on fine-grained biology classification tasks and demonstrates hierarchical representation learning aligned with biological taxonomy.

Digital images and computer vision have become essential tools for studying natural systems across evolutionary biology, ecology, and biodiversity research. Vast quantities of images from museums, camera traps, and citizen science platforms can be converted into actionable biological information including species classification, individual identification, and trait detection. However, applying computer vision to biological questions remains burdensome: researchers must manually label sufficient species-specific data and identify appropriate models for each task. The authors propose that an analogous vision foundation model for biology should be useful for tasks spanning the entire tree of life, instead of just the taxa it has been trained on, significantly lowering barriers to AI adoption in biology.

The authors identify three key design criteria for such a foundation model. First, generalization across the tree of life: the model should support researchers studying many different organisms and generalize to taxa absent from training data. Second, fine-grained representation learning: biology frequently requires distinguishing visually similar species within the same genus or species using mimicry, necessitating fine-grained granularity. Third, strong performance in low-data regimes: due to the expensive nature of biological data collection and labeling, achieving strong results with zero-shot or few-shot learning is critical. Existing work falls short because current datasets lack either scale, diversity, or fine-grained labels, and pre-training strategies insufficiently leverage the tree of life taxonomy.

To address these challenges, the authors present two primary contributions. First, they curate TreeOfLife-10M, which integrates three major sources: iNat21 (2.7M images, 10K species), Bioscan-1M (1.1M insect images), and Encyclopedia of Life (6.6M newly curated images, 448K taxa), resulting in 10.4 million images across more than 454,000 unique taxonomic names. Second, they develop BioCLIP, which repurposes the CLIP multimodal contrastive learning objective to encode hierarchical taxonomic structure into visual representations. The resulting model achieves an average absolute improvement of 18% in zero-shot classification across 10 diverse biology benchmarks.

The authors note that the largest ML-ready biology image dataset is iNat21, which contains 2.7M images of 10K species. Given that the International Union for Conservation of Nature (IUCN) reported over 2 million total described species in 2022, with over 10K bird species and over 10K reptile species alone, existing datasets provide insufficient species diversity for foundation model pre-training. To overcome this, the dataset integrates three complementary sources: iNat21 (2.7 million images, 10,000 species from iNaturalist), Bioscan-1M (1.1 million insect images covering 7,831 families to capture insect diversity), and Encyclopedia of Life (EOL) (6.6 million newly curated images spanning 448,910 taxa).

The authors acknowledge that "taxonomic hierarchies are notoriously noisy and rarely consistent between sources," which has historically contributed to difficulties in creating large-scale biology datasets. The team unified taxonomies across sources using EOL, the Integrated Taxonomic Information System (ITIS), and iNaturalist with special consideration for homonyms — cases where the same name refers to different organisms in different kingdoms. Through this curation process, they achieved 84% full taxa labeling for images in TreeOfLife-10M. The final dataset contains 10.4 million images across more than 450,000 unique taxonomic names and will be released on Hugging Face with accompanying metadata and scripts.

Rather than standard supervised classification, the authors employ the multimodal contrastive learning objective used in CLIP to leverage the hierarchical structure of the label space. Their reasoning emphasizes that hierarchical taxonomy encodes rich biological signal: "if the label space's structure is successfully encoded in a foundation model, even if the model has not seen a certain species, it will likely have learned a good representation for that species' corresponding genus or family." This application is "novel and non-trivial" because TreeOfLife-10M primarily contains class labels rather than free-form text captions, yet the autoregressive text encoder naturally embeds the taxonomic hierarchy into a dense label space.

The model architecture uses ViT-B/16 (Vision Transformer) as the vision encoder and a 77-token causal autoregressive transformer as the text encoder, initialized from OpenAI's CLIP checkpoint. Training is performed with continued pre-training for 100 epochs using a cosine learning rate schedule on 8 NVIDIA A100-80GB GPUs across 2 nodes with a global batch size of 32,768. This continued pre-training strategy — rather than training from scratch — preserves the general visual understanding from CLIP while specializing toward biological domain knowledge.

BioCLIP considers five text representations for pairing with images: (1) Taxonomic name — flattening the taxonomy by concatenating all labels from root to leaf (e.g., "Animalia Chordata Aves Passeriformes Corvidae Pica hudsonia"); (2) Scientific name — genus and species only; (3) Common name — more widespread than Latin names; (4) Scientific + Common; and (5) Taxonomic + Common. The authors propose a "mixed text type training strategy: at each training step, we pair each input image with a text randomly sampled from all of its available text types." This approach "retains the generalization benefits of taxonomic names while providing more flexibility in using other names at inference time."

Evaluation is conducted on 10 diverse benchmark datasets spanning animals, plants, and fungi. For animals: Birds 525 (89,885 images, 525 classes), Plankton (4,080 images, 102 classes), Insects (4,680 images, 117 classes), and Insects 2 (4,080 images, 102 classes). For plants and fungi: PlantNet (1,000 images, 252 classes), Fungi (1,000 images, 252 classes), PlantVillage (1,520 images, 38 classes), Medicinal Leaf (1,040 images, 26 classes), and PlantDoc (1,080 images, 27 classes). Notably, the evaluation includes Rare Species: 12,000 images of 400 IUCN Red List threatened species completely excluded from training, specifically designed to test generalization to unseen taxa.

Zero-shot classification follows the CLIP evaluation procedure. BioCLIP achieves dramatic improvements over both CLIP and OpenCLIP baselines: Birds 525 — 72.1% vs. CLIP 49.9% (+22.2%); PlantNet — 91.4% vs. CLIP 58.5% (+32.9%); Insects — 34.8% vs. CLIP 9.1% (+25.7%); Fungi — 40.7% vs. CLIP 10.2% (+30.5%). Even on the challenging Rare Species benchmark containing 400 species completely absent from training, BioCLIP achieves 37.8% compared to CLIP's 26.6% and OpenCLIP's 31.0%. The mean improvement across all 10 datasets is +18.0% absolute over CLIP, while OpenCLIP actually shows a slight decline of -0.8%, demonstrating that generic large-scale pre-training alone does not suffice for biological tasks.

Few-shot evaluation uses the SimpleShot nearest-centroid classifier, where k examples per class are randomly sampled, centroid averages computed, and classification performed via nearest centroid matching. All experiments are repeated 5 times with different random seeds. In the one-shot setting, BioCLIP achieves 50.4% mean accuracy versus 33.6% for CLIP (+16.8% improvement). Notably, "BioCLIP's mean one-shot accuracy is 9.1% higher than its zero-shot accuracy," contrasting with CLIP's typical pattern where few-shot underperforms zero-shot. In the five-shot setting, BioCLIP reaches 68.9% mean accuracy versus CLIP's 51.5% (+17.4%), confirming that the learned representations are "useful even with only one labeled example."

An ablation study comparing three training objectives on a 1M-image subset of TreeOfLife-10M reveals the critical importance of the CLIP objective. Cross-entropy achieves only 16.7% one-shot and 26.3% five-shot accuracy; hierarchical cross-entropy improves slightly to 19.3% and 30.7%. In stark contrast, the CLIP objective reaches 45.1% one-shot and 64.2% five-shot accuracy. The authors conclude that "the CLIP objective massively outperforms both baselines and strongly justifies our repurposing of the CLIP objective" for structured taxonomic labels. Furthermore, experiments demonstrate that using 1M examples from TreeOfLife-10M outperforms using 2.7M examples from iNat21, highlighting the importance of dataset diversity over raw scale.

t-SNE visualization analysis reveals that BioCLIP preserves taxonomic hierarchy far better than CLIP. At the Kingdom level, both models separate categories cleanly. However, at finer granularities the differences become striking: "only BioCLIP successfully separates the orders in the Insecta Class," and "only BioCLIP cleanly separates families within the Lepidoptera Order" (butterflies and moths). This progressive separation across taxonomic ranks — from kingdom to order to family — demonstrates that BioCLIP "has learned a more fine-grained hierarchical representation conforming to the tree of life, explaining its superior generalization" to unseen taxa.

The paper positions BioCLIP within three research areas. In multimodal foundation models, it references CLIP, ALIGN, and BASIC, noting recent work shows that "dataset diversity and better alignment between image and caption semantics are more important than dataset size." In domain-specific CLIP models, it points to specialized computational pathology CLIP models gathering 111M+ image-text pairs, emphasizing that TreeOfLife-10M provides comparable scale with focus on species diversity. In hierarchy in computer vision, it discusses prior hierarchical classification approaches, noting that prior work "applied hierarchies to smaller label spaces" while BioCLIP handles 450K unique labels — orders of magnitude larger.

Among domain-specific vision models, iNaturalist-pretrained models represent the closest prior art. However, these models are typically trained with standard cross-entropy on a closed label set and cannot generalize to unseen species at inference time. BioCLIP's use of contrastive learning with text-based taxonomic labels fundamentally changes the inference paradigm: any species can be recognized at test time simply by providing its taxonomic name as text, without retraining. This open-vocabulary capability, combined with the scale of TreeOfLife-10M, positions BioCLIP as the first true vision foundation model for organismal biology.

The authors summarize their dual contributions: TreeOfLife-10M and BioCLIP — "a large-scale diverse biology image dataset and a foundation model for the tree of life, respectively." The paper demonstrates strong performance in zero-shot and few-shot settings with evidence that BioCLIP "has learned useful visual representations that are useful even with only one labeled example." Looking ahead, the authors envision "further scaling up the data, e.g., incorporating research-grade images from iNaturalist.org with 100M+ images, and collecting richer textual descriptions" to move closer to the full diversity of the tree of life.