Large language models have demonstrated substantial advancements in reasoning capabilities, particularly through inference-time scaling, as illustrated by models such as OpenAI's o1. However, current Vision-Language Models (VLMs) often struggle to perform systematic and structured reasoning, especially when handling complex visual question-answering tasks. In this work, we introduce LLaVA-CoT, a novel VLM designed to conduct autonomous multistage reasoning. Unlike chain-of-thought prompting, LLaVA-CoT independently engages in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation. We compile the LLaVA-CoT-100k dataset with structured reasoning annotations and propose an inference-time stage-level beam search method. Remarkably, with only 100k training samples, LLaVA-CoT outperforms its base model by 8.9% on multimodal reasoning benchmarks, and surpasses larger models including Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct.

Recent advances in inference-time scaling have shown that allowing models more "thinking time" leads to better reasoning. OpenAI's o1 model exemplifies this by performing extended chain-of-thought (CoT) reasoning before answering. However, simply applying CoT prompting to existing VLMs yields inconsistent improvements — the models often produce unstructured or hallucinated reasoning chains. The key insight is that effective visual reasoning requires not just more tokens, but structured reasoning organized into distinct cognitive stages.

We propose LLaVA-CoT, which structures the reasoning process into four explicit stages: (1) Summary — outlining the problem and determining the approach; (2) Caption — extracting and interpreting relevant visual information from images; (3) Reasoning — proceeding step-by-step through logical problem-solving; (4) Conclusion — delivering the final answer with supporting evidence. Each stage is demarcated by special tokens in the training data, enabling the model to autonomously transition between stages without external prompting.

Chain-of-thought reasoning has proven effective in large language models, with approaches ranging from few-shot CoT prompting to zero-shot "Let's think step by step." Vision-Language Models like LLaVA, InternVL, and Qwen-VL have shown strong performance on visual understanding tasks. However, attempts to integrate CoT into VLMs have been limited to prompting-based approaches that do not fundamentally change the model's reasoning behavior. Inference-time scaling methods like beam search and self-consistency have shown promise, but applying them uniformly across all tokens is computationally wasteful. Our stage-level beam search (SWIRES) method addresses this by allocating more search budget to reasoning-critical stages.

LLaVA-CoT is built on the Llama-3.2-11B-Vision-Instruct backbone. The model is fine-tuned on the LLaVA-CoT-100k dataset with structured reasoning annotations organized into four stages using special delimiter tokens: <SUMMARY>, <CAPTION>, <REASONING>, and <CONCLUSION>. During training, the model learns to generate these stage transitions autonomously, so at inference time it naturally produces structured multi-stage reasoning without any special prompting. The training data is compiled from various visual question answering sources, with structured reasoning annotations generated by distilling reasoning traces from GPT-4o.

The LLaVA-CoT-100k dataset contains 100,000 samples sourced from diverse visual question answering benchmarks. Each sample includes an image, a question, and a structured reasoning annotation with four explicitly delineated stages. The annotations are generated by prompting GPT-4o to produce step-by-step reasoning organized into the summary, caption, reasoning, and conclusion format. Quality filtering ensures that only samples where the GPT-4o reasoning leads to the correct final answer are retained. This curation process ensures high-quality structured reasoning supervision that captures the deliberate cognitive process rather than just the final answer.

We propose SWIRES (Stage-Wise Retracing Search), an inference-time scaling method that operates at the stage level rather than the token level. Instead of generating multiple complete reasoning chains and selecting the best, SWIRES generates multiple candidates at each reasoning stage, evaluates them, and selects the best candidate before proceeding to the next stage. This approach is more efficient than token-level beam search because it concentrates search budget on the stages where reasoning quality matters most (typically the Reasoning stage), while spending fewer resources on stages where the output is more deterministic (like Conclusion).

LLaVA-CoT is evaluated across six challenging multimodal reasoning benchmarks. The model outperforms its base model Llama-3.2-11B-Vision-Instruct by 8.9% on average. Remarkably, the 11B-parameter LLaVA-CoT surpasses Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct — models that are significantly larger or proprietary. The SWIRES inference method provides additional gains beyond standard greedy decoding, demonstrating effective inference-time scaling. Ablation studies show that all four stages contribute to performance, with the Reasoning stage being most critical, and that structured annotations outperform unstructured CoT annotations on the same base data.

LLaVA-CoT demonstrates that structured multistage reasoning can dramatically improve the reasoning capabilities of vision-language models without increasing model size. By organizing reasoning into summary, caption, reasoning, and conclusion stages and training on only 100k carefully curated samples, a relatively small 11B model can rival or exceed much larger systems. The SWIRES inference strategy further leverages this structure for efficient inference-time scaling. This work suggests that how models think matters as much as how large they are.