Modular Caption-then-Reason Approach

Updated 13 January 2026

Modular caption-then-reason approach is a design paradigm that splits image captioning from logical reasoning to boost interpretability and scalability.
It features explicit modular interfaces that decouple perceptual processing from inference, enabling targeted debugging and plug-and-play upgrades.
Empirical studies show that these pipelines yield robust performance in multi-modal tasks, mitigating issues from data scarcity and distribution shifts.

A modular caption-then-reason approach defines a family of architectures in which perceptual understanding (captioning or structured description generation) is explicitly decoupled from downstream reasoning or language behavior. Pioneered in both image and multi-modal reasoning settings, this methodology embeds intermediate symbolic representations between perception and cognition, offering increased interpretability, modular scalability, targeted debugging, and improved robustness under distribution shift or data scarcity. It stands in explicit contrast to monolithic, end-to-end architectures where visual and linguistic reasoning are blended into a single latent sequence.

1. Modular Decomposition and Architectural Principles

The modular caption-then-reason paradigm splits the vision–language pipeline into (at least) two sequential modules:

A captioner (or feature extractor) that produces a structured or free-text description from visual input, often conditioned on the context or downstream query.
A reasoner (LLM or neural module network) that consumes these textual/perceptual summaries (with or without the raw image) to perform logical inference, question answering, generation, or multi-step reasoning.

Representative instantiations include CVLNM for image captioning (Yang et al., 2022), CapGeo for geometric diagram reasoning (Li et al., 10 Oct 2025), CapPO and RACRO for math/logic VQA (Tu et al., 26 Sep 2025, Gou et al., 5 Jun 2025), and FlexCap for dense VQA with regional control (Dwibedi et al., 2024).

Key design principles:

Explicit modular interfaces: Each module exposes a stable text or symbolic interface, allowing plug-and-play replacement, independent scaling, or focused pretraining.
Separation of “what,” “how,” and “why”: Visual modules encode salient entities and relations (“what”), linguistic controllers mediate assembly (“how”), and a reasoning/fact module supplies context or commonsense (“why”) (Yang et al., 2022).
Intermediate symbolic representations: The caption serves as a form of “verbal working memory,” enabling the reasoner to carry out abstract symbolic manipulations divorced from raw perceptual tokens (Weng et al., 24 May 2025).

2. Algorithmic Realizations and Module Specialization

Modular pipelines leverage specialized subtasks and loss functions for perception and reasoning. Table 1 summarizes key module types and integration mechanisms from leading systems.

System	Perceptual Front-End	Reasoner Back-End	Integration Mechanism
CVLNM	4 visual-linguistic modules (noun, adj, verb, function)	Memory module (ConceptNet) + GRU decoder	Self-attn controller + POS syntax loss
CapPO/RACRO	Frozen captioner (caption->text)	RL-optimized VLM (Qwen2.5-VL-7B)	KL-reg, reward optimization
CapGeo	Vision+LLM (captioner)	LLM (Claude-Opus, Qwen)	Composed prompt, keypoint evaluation
FlexCap	Length- and region-controlled ViT captioner	PaLM2-S LLM (instruction-tuned)	Structured prompt batched by region
RMN (video)	Locate/Relate/Func reasoning modules	LSTM-based auto-regressive decoder	Gumbel-softmax POS-tag guidance

Architecture details:

CVLNM (Yang et al., 2022) uses dynamically collocated visual-linguistic modules (noun/adjective/verb/function-word) controlled by a multi-head self-attention gating network, regularized by a part-of-speech syntax loss to align selection weights with target POS tags.
CapPO (Tu et al., 26 Sep 2025) implements an RL policy fine-tuning framework, regularized by the KL divergence between outputs conditioned on images and outputs conditioned on generated captions, adaptively weighting rollouts to favor perceptual consistency.
RACRO (Gou et al., 5 Jun 2025) directly reinforces the caption generator using feedback from reasoning outcome correctness (as determined by a frozen LLM reasoner), closing the perception-reasoning loop via reward-optimized learning.
CapGeo (Li et al., 10 Oct 2025) leverages a keypoint-based metric (elements, spatial relations, numerical attributes) to evaluate caption faithfulness and maximize downstream reasoning utility in geometry QA.

3. Training Objectives and Losses

Statistically principled supervision of modular caption-then-reason pipelines decomposes the global loss into specialized components:

Captioning loss: Standard cross-entropy over ground-truth caption sequences, sometimes conditioned on region, input box, or context (Dwibedi et al., 2024).
Reasoning loss: Cross-entropy or RL policy loss over the answer distribution, conditioned on the generated caption and the task prompt (Gou et al., 5 Jun 2025, Tu et al., 26 Sep 2025, Li et al., 10 Oct 2025).
Consistency regularization: In CapPO, a KL divergence is imposed between the model’s output conditioned on image vs caption to enforce grounding (Tu et al., 26 Sep 2025).
Syntax and layout loss: In CVLNM and RMN, module selection is regularized via POS- or syntax-style cross-entropy for interpretable alignment (Yang et al., 2022, Tan et al., 2020).
Reward shaping: RACRO’s RL objective (see below) rewards captions that maximize downstream correctness as judged by a frozen reasoning LLM:

$J(\theta) = \mathbb{E}_{c\sim\pi_\theta}[R(c;I,q)]$

where $R$ is binary answer correctness (Gou et al., 5 Jun 2025).

These multi-term objectives enable targeted learning: perception can be explicitly regularized for informativeness, faithfulness (to the image), and utility (for the reasoner). Captioner modules can be frozen for backward compatibility, and reasoning modules swapped for future LLM upgrades (Gou et al., 5 Jun 2025).

4. Empirical Performance and Ablation Findings

Extensive benchmark studies on image captioning, VQA, math/logic VQA, geometric reasoning, dense captioning, and video captioning demonstrate both accuracy improvements and improved robustness.

CVLNM (Yang et al., 2022) achieves 129.5 CIDEr-D (vs 124.8 for monolithic SOTA) and maintains performance ( < 3 CIDEr drop) under data scarcity, while Transformer baselines drop >7 CIDEr. Module selection layout accuracy exceeds 92%.
CapPO (Tu et al., 26 Sep 2025), on Qwen2.5-VL-7B, gains +6.0% accuracy in math (44.8→50.8) and +2.4% in general reasoning (59.5→61.9). Ablation studies show KL-regularization and advantage-reweighting are each necessary for full gains; perception-induced errors in reasoning drop by 5.4 pp.
RACRO (Gou et al., 5 Jun 2025) improves MathVision accuracy from 42.0% to 48.7%, and is able to “plug-and-play” new LLMs for further gains (e.g., +3.8% with Qwen3-8B at inference, no retraining). Caption optimization (CRO) adds +3–5% over baseline.
CapGeo (Li et al., 10 Oct 2025) in geometric VQA, using captions, boosts Qwen2.5-VL-72B accuracy from 8.6% to 59.0% and Claude-Opus-4 from 44.8% to 73.0%, with keypoint recall in captions strongly predicting downstream accuracy.
FlexCap (Dwibedi et al., 2024) enables state-of-the-art zero-shot performance for dense captioning (46.9 mAP on Visual Genome) and competitive VQA numbers (OK-VQA: 52.1%, VizWiz: 37.1%).

Ablative results consistently confirm that the intermediate caption bottleneck both identifies the locus of model failure (perception vs reasoning) and can be modulated independently.

5. Analysis, Limitations, and Robustness

The modular caption-then-reason approach offers interpretability, modularity, and robustness, but reveals nuanced failure modes:

Interpretability: Intermediate captions and module activations permit transparent, token-aligned debugging.
Generalization: Decoupling allows adaptation of new reasoning models without retraining the perception front-end (Gou et al., 5 Jun 2025).
Failure sources: For geometric/math QA, captions may omit critical numeric or spatial constraints; reasoning on incomplete captions offers only marginal improvements (Singh et al., 2024).
Prompt structure: In multi-turn or task-based VQA, task-specific prompting and reasoning scaffolds (e.g., chain-of-thought with explicit “Approach” sections) generally outperform simplistic caption-then-answer pipelines, especially on math-heavy tasks where captions lack sufficient detail (Singh et al., 2024).
Scaling: Caption-then-reason gains shrink as base perception quality or LLM scale improves; high-fidelity captions subsume nearly all needed diagram content, but at the cost of greater compute and sometimes redundancy.

6. Broader Applications and Extensions

The modular caption-then-reason paradigm has been generalized or extended in several ways:

Visual-linguistic neural module networks: CVLNM and RMN explicitly model module selection dynamics and can be extended to VQA and video tasks (Yang et al., 2022, Tan et al., 2020).
Task-specific captioning and reasoning: Task-based prompting, chain-of-thought guided pipelines, and region-adaptive captioning (e.g., FlexCap) enable fine-grained control over both perception and reasoning (Dwibedi et al., 2024, Singh et al., 2024).
Ranking, regret, and policy optimization: EGRM for sarcasm generation uses a multi-stage caption, candidate synthesis, and ranking by visual, semantic, and fluency factors (Ruan et al., 2022).
Plug-and-play vision-language reasoning: CapPO and RACRO frameworks allow swapping in new LLMs for downstream reasoning with backward-compatible perception, scaling with LLM advances (Tu et al., 26 Sep 2025, Gou et al., 5 Jun 2025).

Empirical findings suggest that modular caption-then-reason pipelines are particularly advantageous in domains with:

Bottlenecks in visual grounding (geometry, spatial reasoning, fine-grained attention).
Need for interpretability or explainability.
Scenarios requiring cross-model or cross-modal transfer and scalability.

7. Representative Algorithms and Technical Recipes

The modular caption-then-reason workflow is most concisely illustrated by two canonical forms:

CVLNM (Yang et al., 2022): at each time step $t$ , a self-attention controller computes weighting $g_{t,k}$ over four specialized modules; the outputs are fused, passed through a memory-based reasoner, and the next token is predicted. The learning objective combines captioning, POS-syntax, and layout losses.
RACRO (Gou et al., 5 Jun 2025): the captioner is trained with RL, directly optimizing for downstream answer correctness via a surrogate loss that mixes PPO-style clipped ratios and KL regularization.
CapPO (Tu et al., 26 Sep 2025): policy updates are weighted by the KL-divergence between image- and caption-conditioned outputs, emphasizing perceptual consistency at every trajectory.
CapGeo (Li et al., 10 Oct 2025): captions are generated (prompt-driven); a reasoner LLM consumes (question, image, caption) and produces the answer; keypoint-based metrics automate caption evaluation.

Canonical pseudocode for a two-stage Caption-then-Reason pipeline, as exemplified in (Weng et al., 24 May 2025):

captions = []
for I_t in image_sequence:
    cap_t = Captioner(I_t)  # e.g., "red square in top right"
    captions.append(cap_t)

answer = Reasoner(question, captions)  # Reasoner sees only text, not images

References

Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning (Yang et al., 2022)
Perceptual Decoupling for Scalable Multi-modal Reasoning via Reward-Optimized Captioning (Gou et al., 5 Jun 2025)
Caption This, Reason That: VLMs Caught in the Middle (Weng et al., 24 May 2025)
Beyond Captioning: Task-Specific Prompting for Improved VLM Performance in Mathematical Reasoning (Singh et al., 2024)
Perception-Consistency Multimodal LLMs Reasoning via Caption-Regularized Policy Optimization (Tu et al., 26 Sep 2025)
FlexCap: Describe Anything in Images in Controllable Detail (Dwibedi et al., 2024)
CapGeo: A Caption-Assisted Approach to Geometric Reasoning (Li et al., 10 Oct 2025)
Learning to Discretely Compose Reasoning Module Networks for Video Captioning (Tan et al., 2020)
How to Describe Images in a More Funny Way? Towards a Modular Approach to Cross-Modal Sarcasm Generation (Ruan et al., 2022)