Today's most advanced multimodal models remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed models into open ones. As a result, the community is still missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a family of open vision-language models that achieves state-of-the-art performance among open-weight and open-data models. Our key innovation is a novel, human-annotated image caption dataset created using speech-based descriptions. We also introduce diverse fine-tuning datasets including in-the-wild Q&A and 2D pointing data.

The best-in-class 72B model within the Molmo family not only outperforms others in the class of open weight and data models but also compares favorably against proprietary systems like GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human evaluation. Even the smallest model, MolmoE-1B, nearly matches the performance of GPT-4V. All model weights, caption and fine-tuning data, and source code are released to the public.

Extensions to large language models that process images alongside text have achieved impressive multimodal capabilities. The most performant of these vision-language models (VLMs), however, remain proprietary with neither model weights, data, nor code being publicly released. Researchers have pursued open VLM development, with early works like LLaVA producing fully open weights and training data but now lagging significantly behind state-of-the-art performance. More recent, stronger open-weight models have trended towards less open data: the training data may either be proprietary, or there is a heavy reliance on synthetic data generated by proprietary systems.

The resulting VLMs are effectively distillations of proprietary VLMs, and the scientific community is still missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a family of state-of-the-art open VLMs. This result is achieved with a simple training pipeline in which we connect an independently pre-trained, off-the-shelf vision encoder and language model and jointly train the resulting VLM to generate captions from a newly collected dataset of detailed, high-quality, dense image descriptions. A critical challenge in dense caption collection is that if asked to write an image description, the result often only mentions a few salient visual elements; if a minimum word count is enforced, annotators will either take too long or copy-and-paste responses from proprietary VLMs.

Our innovation leverages a shift in modality: we ask annotators to describe images in speech for 60 to 90 seconds rather than asking them to write descriptions. With this modality switching "trick," annotators provide far more detailed descriptions in less time. The annotators' audio is then transcribed using an off-the-shelf speech-to-text system, and the transcribed text is processed using a language-only LLM to improve text quality, such as removing spoken artifacts and normalizing style. This process produces dense, detailed image captions that rival or exceed captions generated by proprietary VLMs, while being genuinely human-authored.

Our model architecture follows the simple and standard design of combining a language model with a vision encoder. It consists of four components: (1) a pre-processor that converts the input image into a set of multiscale, multi-crop images, (2) a ViT image encoder that independently maps each of these images into a set of vision tokens, (3) a connector that projects the vision tokens to the language model's input dimension with an MLP and then pools the vision tokens to reduce their count, and (4) a decoder-only Transformer LLM. From this template, we construct a family of models parameterized by the choice of vision encoder and LLM.

The released models employ OpenAI's ViT-L/14 336px CLIP model as the vision encoder, which provides consistently good results. For the language model component, we explore several options: OLMo-7B-1024 (fully open), OLMoE-1B-7B (a mixture-of-experts variant with only 1B active parameters), Qwen2 7B, and Qwen2 72B. Given these choices, the subsequent training data and recipe are the same for all models aside from optimizer learning rates. This design allows us to study the effect of language model scale and architecture while keeping all other variables constant.

The training process consists of two stages: multimodal pre-training for caption generation and supervised fine-tuning. All model parameters are updated in both stages. We do not use RLHF. For the first stage, we started by sourcing web images according to a diverse set of approximately 70 high-level topics (e.g., street signs, memes, food, drawings, websites, blurry photos), and for each image we asked three annotators to describe the image in detail by speaking for at least 60 seconds. Annotators received prompts addressing: image overview, object counts, text content, spatial positioning, subtle details, background elements, and style or color.

The annotators' audio was then transcribed using an off-the-shelf speech-to-text system, and the transcribed text was processed using a language-only LLM to improve the text quality, for example removing spoken artifacts and normalizing style. Our training process uses all four of these image LLM-processed transcripts, when available, as a form of naturalistic data augmentation. In total, we trained on 712k distinct images with approximately 1.3M captions including the augmentation. This represents a fundamentally different approach from synthetic caption generation: the visual observations are genuinely human-sourced, with the LLM serving only as a stylistic normalizer rather than as the source of visual understanding.

After training for captioning, we fine-tune all model parameters on a mixture of supervised training data. This mixture includes common academic datasets and several new PixMo datasets. PixMo-AskModelAnything was created with 162k question-answer pairs from 73k images, enabling diverse user query handling through an annotation workflow where an annotator would select an image, write a question about it, and iterate with model responses until the answer was acceptable. PixMo-CapQA generated 214k question-answer pairs from 165k images by prompting a language-only LLM given ground-truth captions. PixMo-Docs created 2.3M question-answer pairs from 255k document, chart, and table images using code generation and privileged access to underlying data.

Additional fine-tuning data includes PixMo-Clocks, constructed from 826k examples using synthetic analog clock images with time-reading questions, targeting a specific weakness in VLMs. The academic datasets incorporated include VQA v2, TextVQA, OK-VQA, ChartQA, DocVQA, InfoVQA, AI2D, RealWorldQA, MMMU, and others. This careful mixture of human-annotated, programmatically generated, and academic datasets ensures that the model develops broad competence across diverse visual understanding tasks while maintaining strong general conversational abilities.

We collected pointing data that achieves three goals: (1) enables the model to point to anything described by text, (2) enables the model to count by pointing, and (3) enables the model to use pointing as a natural form of visual explanation when answering questions. The collection process involved asking human annotators to point at something in an image, write a description of it, and then point to every instance of it in the image, making the pointing exhaustive. The resulting PixMo-Points dataset contains 2.3M question-point pairs from 428k images, plus an additional 79k question-answer pairs from 29k images for explanation-based pointing.

This pointing capability represents a distinctive feature of Molmo that sets it apart from most existing VLMs. Rather than requiring specialized object detection heads or bounding box regression modules, Molmo performs pointing by generating normalized (x, y) coordinates as text tokens within its standard language modeling framework. This design choice means that the same model architecture handles both natural language generation and spatial grounding, unifying visual understanding and localization into a single, coherent interface without any task-specific architectural modifications.

Vision-language model evaluation is evolving rapidly, with new academic benchmarks constantly appearing. As a result, academic benchmarks provide only a partial picture of how a model performs. We evaluate across 11 academic benchmarks including AI2D, ChartQA, VQA v2, DocVQA, InfoVQA, TextVQA, RealWorldQA, MMMU, MathVista, CountBenchQA, and Flickr Count. To complement these benchmarks, we perform a human evaluation that allows us to rank models according to user preference. We collected a diverse set of 15k image and text prompt pairs and queried a set of VLMs for responses, then collected greater than 325k preference ratings (approximately 450 matches per model pair) from approximately 870 human annotators.

Key results demonstrate the effectiveness of our approach across model scales. MolmoE-1B, with only 1B active parameters, nearly matches the performance of GPT-4V on academic benchmarks. Molmo-7B-D (based on Qwen2 7B) achieves an average academic benchmark score of 77.3, performing between GPT-4V and GPT-4o. The flagship Molmo-72B achieves the highest academic benchmark score of 81.2 and ranks second in Elo rating, just behind GPT-4o, while outperforming Gemini 1.5 Pro, Gemini 1.5 Flash, and Claude 3.5 Sonnet. On individual benchmarks, Molmo-72B reaches 96.3 on AI2D, 87.3 on ChartQA, 93.5 on DocVQA, and 81.9 on InfoVQA.

From the preference rankings, we calculated an Elo ranking using the Bradley-Terry model across 27 models. The human evaluation reveals several noteworthy findings. First, model ranking on academic benchmarks does not perfectly correlate with human preference — some models that score well on benchmarks are rated lower by human evaluators, and vice versa. Second, Molmo models consistently rank higher in human evaluation than their academic benchmark scores would suggest, indicating that our training data produces models with strong real-world conversational utility beyond narrow benchmark performance. This suggests that PixMo's human-authored captions may endow models with more natural, user-aligned response patterns.

We have presented Molmo, a family of open vision-language models that demonstrates state-of-the-art performance can be achieved with fully open weights and open data, without reliance on synthetic data from proprietary systems. Our key contribution is the PixMo data ecosystem, anchored by the insight that speech-based image description elicits far richer and more detailed visual annotations than traditional written approaches. By combining this novel caption collection methodology with carefully curated fine-tuning datasets for question answering, document understanding, and spatial pointing, we produce models that are competitive with leading proprietary VLMs across both academic benchmarks and human preference evaluation.

Looking forward, our work provides foundational knowledge for the community about how to build performant VLMs from scratch. The speech-based annotation paradigm opens new possibilities for scaling high-quality data collection across diverse languages and domains. The pointing capability suggests a path toward more grounded and interpretable multimodal AI systems, where models can not only describe what they see but also precisely indicate where they see it. By releasing all model weights, data, and code, we aim to accelerate progress toward truly open and reproducible multimodal AI research.