Vision-DeepResearch Paradigm

• Vision-DeepResearch is a multimodal deep research framework that enables multi-turn, multi-entity visual and textual evidence aggregation in complex scenarios.
• It integrates specialized architectures—combining vision transformers and language models—with tool-call policies to drive iterative, agentic reasoning.
• Robust benchmarks such as VDR-Bench and combined supervised and reinforcement learning pipelines validate its high accuracy and real-world robustness.

Vision-DeepResearch denotes a new paradigm and set of benchmarks, algorithms, and architectures for instilling, measuring, and advancing the deep-research capabilities of multimodal LLMs (MLLMs). It specifically targets scenarios in which end-to-end agents must perform complex, multi-step visual and textual information search, aggregation, and reasoning on real-world input images and questions—robustly overcoming visual noise, ambiguity, and the insufficiency of internal model knowledge. This approach has produced a suite of agentic MLLMs, dedicated evaluation resources, and training pipelines that enable, measure, and benchmark high-fidelity, iterative multimodal fact-finding systems (Huang et al., 29 Jan 2026, Zeng et al., 2 Feb 2026).

1. Multimodal Deep-Research Paradigm: Principles and Formal Definition

Vision-DeepResearch establishes that practical multimodal fact-finding must move beyond single-pass VQA pipelines or naïve "tool-augmented" workflows. It formalizes the retrieval process as a multi-turn, multi-entity, and multi-scale search paradigm:

• Multi-turn: The MLLM-agent proceeds iteratively, at each step making decisions about which visual region(s) or textual queries to submit to external engines (e.g., web search, image search, or other tools), integrating new evidence before proceeding.
• Multi-entity, multi-scale: For each image and question, the agent proposes a set $S_t = \{\Box_t^{(e,s)}\}$ of region proposals over entities $e$ and spatial scales $s$, enabling targeted, fine-grained evidence acquisition.
• Trajectories: The agent's full reasoning trajectory comprises tuples $(R_t, A_t, O_t)$—reasoning, tool call, and observations—stored and used for subsequent reasoning and answer justification.

Letting $T$ denote the maximum number of tool-interaction turns, the agent's interaction is governed by:

$$A_t^v = \mathrm{VisionSearchTool}(S_t), \quad O_t^v = \mathrm{VisionPipeline}(A_t^v), \quad V_t^v = \{O_1^v, \dots, O_t^v\}$$

$$h_t^v = \mathrm{Judge}(I, q, V_t^v, a^{\text{true}}) \in \{0, 1\}$$

where $h_t^v$ is a hit/miss signal to allow bridge to text-based evidence gathering.

This paradigm is realized in open- and closed-source MLLMs with explicit action policies, observation accumulation, and planning/control modules (Huang et al., 29 Jan 2026).

2. Model Architectures and Integration

Vision-DeepResearch agents are built upon strong base MLLMs (notably, Qwen3-VL), extended in specific directions to accommodate agentic reasoning:

• Visual Encoder: Typically a pretrained vision transformer (e.g., ViT), optionally augmented with a lightweight region-proposal/bounding-box head for multi-entity cropping.
• Language Backbone: Decoder-only transformer capable of multi-modality, equipped with an extended context window (up to 64K tokens) to accommodate lengthy search histories and tool traces.
• Tool-Call Policy: An auxiliary policy head (small MLP) emits distributions over possible tool-calls (e.g., 'vision search', 'text search', 'web visit'), parameterized by the hidden state of the LLM at each "think" step.
• Action Parameterization: Each tool-call includes serialized, structured arguments: bounding boxes for cropping, query strings for web/text search, etc.
• Planning Module: In some implementations (e.g., Skywork-R1V4), a planner head (and parameter head) governs the interleaving of code-executed image operations and search tool interactions, producing complex, multi-step plans and supporting explicit long-horizon reasoning (Zhang et al., 2 Dec 2025).

The agent's token-stream is interleaved with action markers ($<\mathrm{think}>$, $<\mathrm{tool\_call}>$, etc.), enabling seamless integration between internal and external reasoning.

3. Training Regimes: Supervised Pretraining, Cold-Start SFT, and RL

Vision-DeepResearch advocates a comprehensive training pipeline:

• Cold-start Supervised Fine-Tuning (SFT): The base MLLM is supervised on 30K+ manually curated, multi-turn, multimodal trajectories, each with explicit tool-calls, crop coordinates, retrieved evidence, and answers. These trajectories incorporate
  • Multi-entity/multi-scale cropping
  • Text-bridged reasoning (expansion of evidence paths with language-based summaries)
  • Fuzzy (obfuscated) multi-hop VQA
• Reinforcement Learning (RL): On-policy RL with binary rewards (correct answer/incorrect), using LLM-as-judge evaluations. Policy gradients are computed, advantage baselines (leave-one-out mean) stabilize updates, and degenerate episodes are filtered. Training is conducted in BF16 to support large context windows and multi-threaded to increase rollout throughput.
• Ablation and Engineering: Empirically, inclusion of RL after SFT, and the coverage of multi-scale cropping and noisy, multi-entity retrieval in trajectories, is crucial for high accuracy and real-world robustness (Huang et al., 29 Jan 2026, Zeng et al., 2 Feb 2026).

4. Benchmarking and Evaluation: VDR-Bench and Metrics

A dedicated benchmark—VDR-Bench—was created to rigorously assess Vision-DeepResearch agents:

• 2,000 Curated Multi-hop VQA Instances: Each composed to ensure that answers require non-trivial, iterative visual search, region cropping, external retrieval, and multi-hop reasoning. All instances are filtered to preclude text-only shortcuts or answer leakage from image captions or model priors (Zeng et al., 2 Feb 2026).
• Metrics:
  • Answer Accuracy (Acc): Standard match of system prediction to ground-truth.
  • Entity-level Recall (ER): Fraction of gold-standard reasoning entities hit by the agent's search trajectory, judged via LLM-based semantic match. Correlates strongly with answer accuracy (ρ≈0.85).
  • Ablation Protocol: Compares Direct Answer, Whole-Image Search (WIS), Cropped-Image Search (CIS), and CIS+TextSearch settings; shows that only the multi-turn, multi-entity, multi-scale approach achieves competitive accuracy.
• Comparative Results: On VDR-Bench, agentic Vision-DeepResearch models (Qwen3-30B, Vision-DeepResearch-30B) exceed predecessors by 16 points in average accuracy, with ablations confirming the necessity of both search and RL tuning (Huang et al., 29 Jan 2026, Zeng et al., 2 Feb 2026).

Table: VDR-Bench Agentic Ablation

Setting	VDR Acc (%)
Direct Answer	4.8
Whole-Image Search	11.8
WIS + TextSearch	16.0
CIS only	15.4
CIS + TextSearch	37.8

5. Analysis, Limitations, and Extensions

Vision-DeepResearch agents demonstrate improved factual accuracy, robust entity-level reasoning, and effective vision-textual search in real-world conditions. However, extant limitations remain:

• External API Dependency: Reliance on live search and web-crawling introduces latency and potential inconsistency in repeated inference.
• Reward Sparsity: The binary reward does not distinguish partial progress; reward shaping (e.g., intermediate entity hits) is not yet standard.
• Vision-First Necessity: Models sometimes default to text-only response; enforcement of visual interaction via multi-turn forcing is critical (Zeng et al., 2 Feb 2026).
• Scalability and Generalization: Although Vision-DeepResearch supports hundreds of tool-calls and long trajectories, further scaling to complex interactive domains (video, GUI) and addition of richer tools (segmentation, OCR, depth, etc.) is actively researched.
• Dataset and Annotator Limitations: High-quality, long-horizon, multi-modal trajectory annotation remains labor-intensive; automated obfuscation, simulation, and synthetic trajectory augmentation are potential avenues.

6. Impact and Future Directions

The Vision-DeepResearch paradigm provides a robust, extensible framework for benchmarking and building next-generation multimodal agents capable of human-competitive deep research on visual-textual evidence. Its public codebase [https://github.com/Osilly/Vision-DeepResearch] and open benchmarks enable reproducible evaluation and training.

Areas for future investigation include:

• Caching and Tool Training: Integrating retrieval caches, directly optimizing tool-use policies, and joint vision-language-tool learning.
• Reward Shaping: Richer internal rewards that account for intermediate entity hits and knowledge graph coverage.
• Multi-Modal Expansion: Beyond static images, expansion to video, interactive environments, and multi-agent collaboration.
• Hybrid Training: Combining high-quality SFT with lightweight, real-environment RL for continual improvement (Zhang et al., 2 Dec 2025).

By establishing a rigorous standard for multi-step, evidence-aggregating reasoning, Vision-DeepResearch is a core reference point for agentic MLLM research in vision, robust tool-use, and automated scientific and technical search (Huang et al., 29 Jan 2026, Zeng et al., 2 Feb 2026, Zhang et al., 2 Dec 2025).