We present VisProg, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VisProg avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate Python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable visual rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing routines, or Python functions to produce intermediate outputs that may be consumed by subsequent steps. We demonstrate the flexibility of VisProg on four diverse tasks — compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing.

The dominant paradigm in contemporary AI involves building end-to-end trainable models that undergo extensive unsupervised pretraining followed by supervised multitask training. However, this approach demands carefully curated datasets for each task, making it challenging to scale to the "long tail" of complex, compositional tasks. In this work, we investigate an alternative: leveraging large language models to decompose natural language task descriptions into simpler steps that can be executed by specialized models or programs.

VisProg accepts visual data along with a natural language instruction, generates a sequence of computational steps (a visual program), and executes these steps to produce the desired output. Each line of the program invokes a module from a library of pre-existing computer vision models, language models, image processing routines from OpenCV, or arithmetic/logical operators. Modules consume outputs from preceding lines and produce intermediate results for subsequent consumption. This design enables VisProg to flexibly compose diverse capabilities without requiring any task-specific training or fine-tuning.

The system improves upon previous approaches like Neural Module Networks (NMN) in several key ways: it employs GPT-3 for program generation without requiring training, uses higher-level abstractions, and invokes state-of-the-art pre-trained models alongside non-neural Python subroutines. VisProg also exhibits strong interpretability through readable programs that allow users to verify logical correctness and inspect intermediate step outputs for diagnosis and intervention. Key contributions include: (i) a system utilizing large language model in-context learning for visual program generation from natural language instructions; (ii) demonstration of flexibility on complex visual tasks including factual knowledge object tagging and language-guided image editing; and (iii) generation of visual rationales showing utility for error analysis and instruction tuning.

Neural Module Networks pioneered modular, compositional approaches for visual question answering by composing neural modules into differentiable networks. Early attempts employed off-the-shelf parsers, while recent methods learn layout generation jointly with neural modules using reinforcement learning and weak answer supervision. VisProg differentiates itself by generating high-level programs invoking trained state-of-the-art neural models and Python functions at intermediate steps, enabling incorporation of symbolic, non-differentiable modules. It leverages large language models' in-context learning to generate programs via prompting with natural language instructions and similar examples, eliminating the need for specialized task-specific program generators.

Recent work applies LLMs and in-context learning to visual tasks. PICa utilizes LLMs for knowledge-based VQA by representing visual information as text (captions, objects, attributes) and feeding textual representations to GPT-3 alongside questions and in-context examples. Socratic Models compose pre-trained models from different modalities (language, vision-language, audio-language) for zero-shot tasks including image captioning and video-to-text retrieval. Unlike Socratic Models where composition is task-predetermined, VisProg determines per-instance model composition through program generation based on instructions. ProgPrompt, a concurrent work, demonstrates LLM capability for generating Python-like robot action plans from natural language, though VisProg programs accommodate more general inputs including strings, numbers, expressions, and arbitrary Python objects.

A growing literature explores LLM-based language reasoning through prompting. Chain-of-Thought (CoT) prompting demonstrates impressive capabilities for math reasoning by prompting models with input-rationale-output examples. While CoT relies on LLMs generating and executing reasoning paths internally, VisProg similarly employs decomposition approaches where initial decomposition generates sub-tasks handled by specialized sub-task processors. The key distinction is that VisProg externalizes execution to dedicated vision modules rather than relying on the LLM's own reasoning capacity for perceptual tasks.

VisProg exploits GPT-3's remarkable generalization ability from limited input-output demonstrations. Following the translation paradigm, VisProg prompts GPT-3 with instruction-program pairs, enabling visual program generation for novel natural language instructions without fine-tuning. In-context examples feature manually written, high-level programs constructed independently of accompanying images. Each program step comprises a module name, input argument names with values, and an output variable name. Programs typically employ output variables from prior steps as inputs for subsequent steps.

The system uses descriptive naming conventions — module names like "Select" or "ColorPop," argument names like "image" or "query," and variable names like "IMAGE" or "OBJ" — enabling GPT-3 to comprehend input-output types and module functions purely from naming. During execution, output variables store arbitrary data types — for instance, "OBJ" instances contain lists of image objects with associated masks, bounding boxes, and text categories. This flexible type system allows modules to pass rich, structured intermediate representations between steps.

VisProg currently supports 20 modules enabling image understanding, image manipulation including generation, knowledge retrieval, and arithmetic-logical operations. Each module implements a Python class with three core methods: (i) parse — extracting input argument names, values, and output variable names from a program line; (ii) execute — performing necessary computations involving trained neural models while updating the program state; and (iii) html — visually summarizing step computations via HTML for visual rationale creation. Adding new modules requires only implementing and registering a module class; the VisProg interpreter automatically handles program execution.

The module library spans two broad categories. Neural modules leverage state-of-the-art pre-trained models: OWL-ViT for open-vocabulary object localization, DSFD for face detection, MaskFormer for semantic segmentation, CLIP for zero-shot image classification, ViLT for visual question answering, and Stable Diffusion for image generation and inpainting. Non-neural modules include image processing subroutines from OpenCV (cropping, overlaying, color manipulation), arithmetic and logical Python operators (counting, comparison, expression evaluation), and knowledge retrieval via GPT-3 prompting. This hybrid design ensures that each sub-task is handled by the most appropriate tool — neural models for perception, symbolic routines for precise computation.

An interpreter manages program execution by initializing program state (a dictionary mapping variable names to values) with inputs, then stepping through the program line-by-line while invoking appropriate modules with specified inputs. Following each step execution, the program state updates with the step's output name and value. This sequential execution model ensures that each module has access to all previously computed results, enabling complex multi-step reasoning chains where later steps build upon earlier computations.

Beyond computation, each module class implements an HTML method summarizing module inputs and outputs. The interpreter concatenates HTML summaries from all program steps into a visual rationale enabling analysis of logical program correctness and inspection of intermediate outputs. Visual rationales facilitate understanding failure reasons and minimal instruction tweaking for performance improvement. This transparency is a fundamental advantage over end-to-end neural approaches that operate as opaque black boxes, offering users a concrete mechanism for diagnosis, debugging, and iterative refinement.

VisProg demonstrates compositional suitability for the GQA multi-step visual question answering task. Task modules include open vocabulary localization, VQA functionality, image region cropping using bounding box coordinates or spatial prepositions ("above," "left"), box counting, and Python expression evaluation. For example, addressing "Is the small truck to the left or to the right of the people that are wearing helmets?" involves localizing helmet-wearing individuals, cropping left and right regions, checking for small trucks, and returning the appropriate direction. The evaluation creates a GQA subset by randomly sampling approximately 100 diverse question types from balanced validation and test-dev sets.

Zero-shot reasoning on image pairs demonstrates VisProg's ability to solve multi-image tasks using single-image VQA systems without multi-image training. Applied to NLVRv2 (Natural Language Visual Reasoning on image pairs), the system decomposes complex statements into simpler single-image questions and Python expressions involving arithmetic-logical operators. For example, verifying "Both images contain exactly one dog" requires asking each image "How many dogs are in this image?" and evaluating whether both answers equal one. The approach achieves statement verification through image-level question answering using ViLT-vqa and expression evaluation using Python operators.

Factual knowledge object tagging involves identifying image objects whose names require external knowledge — celebrities, politicians, TV show characters, flags, corporate logos, organism species, etc. VisProg leverages GPT-3 as an implicit knowledge base queried through natural language prompts (e.g., "List the main characters on the TV show Big Bang Theory separated by commas"). Generated category lists feed into CLIP image classification modules applied to localization and face detection results. For language-guided image editing, VisProg handles complex instructions like "Hide the face of Daniel Craig with an emoji" (de-identification), "Create a color pop of Daniel Craig and blur the background" (object highlighting), and "Replace Barack Obama with Barack Obama wearing sunglasses" (object replacement requiring identification, masking, and inpainting).

Performance progressively improves with increasing in-context examples on both GQA and NLVRv2 validation sets. Each run randomly selects annotated in-context example subsets based on random seeds. Majority voting across random seeds consistently outperforms average run performance, consistent with Chain-of-Thought reasoning literature findings. NLVRv2 performance saturates with fewer prompts than GQA, likely because NLVRv2 programs require fewer modules and demonstrations than GQA. This suggests that prompt diversity matters more than quantity — covering the space of required module combinations is key to strong program generation.

On GQA, VisProg with a curated 20-example prompt achieves 50.0% accuracy, while the voting strategy across 5 runs with 24 random examples reaches 50.5%. Both configurations outperform ViLT-vqa's 47.8% baseline. The curated prompt matches voting strategy performance while requiring 5x less compute, highlighting the promise of prompt engineering over brute-force scaling. On NLVRv2, VisProg achieves 61.8% accuracy with curated prompts and 62.4% with voting, compared to ViLT-nlvr's fine-tuned upper bound of 76.3%. While several points below the fine-tuned model, this zero-shot performance is achieved without any training on image pairs, using only a single-image VQA model trained on VQAv2.

For factual knowledge object tagging, VisProg achieves 63.7% tagging F1 and 80.6% localization F1 on the original instructions. Through instruction tuning informed by visual rationale inspection, performance improves to 75.7% tagging F1 and 84.9% localization F1 — gains of 12.0 and 4.3 points respectively. Modified instructions provide better localization queries ("kitchen appliance" rather than "item"), more informative knowledge retrieval queries, and specified classification category constraints. For image editing, manual semantic correctness evaluation yields 59.8% accuracy on original instructions, improving to 66.4% after instruction tuning. These results demonstrate that visual rationales serve as a practical tool for human-in-the-loop performance improvement.

Visual rationales enable thorough failure mode analysis. For approximately 100 sampled instances per task, manual rationale inspection reveals error sources. On GQA, incorrect programs affect 16% of samples, suggesting performance improvement through additional in-context examples matching currently-failing instruction types or upgraded high-error modules. For NLVR, improved VQA models could improve performance by up to 24%. For knowledge tagging and image editing, improving "List" and "Select" modules represents major error reduction opportunities. This fine-grained error attribution is uniquely enabled by the modular, interpretable nature of VisProg — an advantage that monolithic end-to-end models fundamentally cannot provide.

VisProg demonstrates that visual programming is a straightforward, effective approach for leveraging large language model reasoning capabilities for complex visual tasks. The system generates highly interpretable visual rationales while demonstrating strong performance across four diverse tasks. By composing off-the-shelf models through LLM-generated programs, VisProg avoids the need for task-specific training while maintaining competitive accuracy. Future investigation of enhanced prompting strategies and novel user feedback incorporation methods for neuro-symbolic systems like VisProg represents exciting directions for next-generation general-purpose vision systems.

More broadly, VisProg represents a paradigm shift from training monolithic models to orchestrating specialized components. As foundation models continue to improve in both language understanding and code generation, the visual programming approach stands to benefit directly — better LLMs will produce better programs, and better vision modules will produce better intermediate results, without requiring any change to the VisProg framework itself. The modular, interpretable, and extensible nature of the system positions it as a flexible substrate for building increasingly capable vision systems through composition rather than end-to-end retraining.