MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents

Abstract: Deep Research Agents (DRAs) generate citation-rich reports via multi-step search and synthesis, yet existing benchmarks mainly target text-only settings or short-form multimodal QA, missing end-to-end multimodal evidence use. We introduce MMDeepResearch-Bench (MMDR-Bench), a benchmark of 140 expert-crafted tasks across 21 domains, where each task provides an image-text bundle to evaluate multimodal understanding and citation-grounded report generation. Compared to prior setups, MMDR-Bench emphasizes report-style synthesis with explicit evidence use, where models must connect visual artifacts to sourced claims and maintain consistency across narrative, citations, and visual references. We further propose a unified, interpretable evaluation pipeline: Formula-LLM Adaptive Evaluation (FLAE) for report quality, Trustworthy Retrieval-Aligned Citation Evaluation (TRACE) for citation-grounded evidence alignment, and Multimodal Support-Aligned Integrity Check (MOSAIC) for text-visual integrity, each producing fine-grained signals that support error diagnosis beyond a single overall score. Experiments across 25 state-of-the-art models reveal systematic trade-offs between generation quality, citation discipline, and multimodal grounding, highlighting that strong prose alone does not guarantee faithful evidence use and that multimodal integrity remains a key bottleneck for deep research agents.

• The paper introduces a standardized benchmark that evaluates DRAs on multimodal integration, rigorous citation, and evidence synthesis.
• It details a multi-stage evaluation pipeline combining FLAE, TRACE, and MOSAIC to diagnose writing quality, evidence grounding, and semantic fidelity.
• Experiments reveal that combining agentic orchestration with robust LLM backbones significantly improves multimodal reasoning and citation accuracy.

MMDeepResearch-Bench: A Standardized Evaluation for Multimodal Deep Research Agents

Motivation and Problem Setting

The development of Deep Research Agents (DRAs) marks a significant advancement in retrieval-augmented, tool-using AI systems tasked with open-ended, evidence-grounded synthesis. However, prior evaluations have mostly targeted text-only settings, isolated short-form multimodal QA, or addressed multimodal interaction without examining the end-to-end integration of textual and visual evidence in authentic research workflows. This omission hampers progress on systems required to interpret heterogeneous multimodal artifacts and synthesize long-form, citation-rich reports whose faithfulness cannot be evaluated with simple gold-label matching.

MMDeepResearch-Bench (MMDR-Bench) is introduced to fill this gap by presenting a comprehensive benchmark and evaluation protocol that assesses not only the language generation and reasoning capacity of DRAs, but also their ability to correctly integrate, interpret, and cite multimodal evidence—specifically, image-text bundles relevant to open-ended research tasks across diverse domains.

Benchmark Composition and Task Design

MMDR-Bench is constructed from 140 expert-crafted tasks spanning 21 distinct domains, ranging from everyday information-seeking (“Daily” regime) to complex, analysis-intensive research scenarios (“Research” regime). Key properties of this benchmark include:

• Multimodal Necessity: Each task comprises a text query and one or more images (e.g., charts, diagrams, screenshots). Successful completion requires cross-modal understanding and integration.
• Citation Grounding: Tasks are iteratively refined to ensure that every substantive factual or analytic claim must map to a specific source (textual or visual), enabling claim-level auditable verification.
• Domain Breadth: Tasks cover a range of domains, including data science, engineering, policy, humanities, business, and interdisciplinary exploratory scenarios. This diversity ensures DRAs are tested on both structured scientific visuals and noisy, in-the-wild images.

Expert curation guarantees the necessity of multimodal evidence and the verifiability of answers, eliminating task leakage through text-only shortcuts.

Unified Modular Evaluation Framework

MMDR-Bench is paired with a robust, multi-stage evaluation pipeline, designed to yield fine-grained diagnostic signals that disentangle sources of failure and enable systematic capability auditing. The pipeline comprises three principal components:

1. FLAE (Formula-LLM Adaptive Evaluation)

FLAE jointly blends formulaic text-derived features (e.g., lexical diversity, section structure, citation compliance) with task-aware LLM judge scores. It evaluates each report on:

• Readability: Clarity and coherence
• Insightfulness: Depth of synthesis and non-trivial analysis
• Structure: Completeness and organizational quality, including citation use and integration of images

Adaptive fusion weights are assigned per-task and per-report, ensuring that evaluation remains consistent but robust across domains with differing reporting norms.

2. TRACE (Trustworthy Retrieval-Aligned Citation Evaluation)

TRACE measures citation faithfulness at the atomic claim level, including:

• Consistency: Are cited claims actually supported by the referenced evidence?
• Coverage: Are the major task components and information needs addressed by supported citations?
• Fidelity: Is the answer faithful to the intent of the prompt and the content of the cited sources?
• Visual Evidence Fidelity (VEF): Introduced as a strict pass/fail constraint. VEF ensures the agent’s interpretation of visual evidence matches a canonical, human-constructed textual ground truth of each image, rigorously penalizing hallucinated, mis-attributed, or omitted visual content.

3. MOSAIC (Multimodal Support-Aligned Integrity Check)

MOSAIC examines the factual and semantic integrity of multimodal claims. It parses reports into discrete “MM-items,” each referencing both a narrative element and a visual artifact. MOSAIC then applies visual-type-specific verification (e.g., data plausibility for charts, semantic grounding for photos), aggregating scores for visual-semantic alignment, data extraction accuracy, and multimodal question answering quality.

A gated evaluation strategy triggers MOSAIC only for reports passing FLAE and TRACE threshold criteria, ensuring computational and auditing efficiency.

Key Findings from Model Benchmarks

Comprehensive experiments assess 25 state-of-the-art models, including text-only and multimodal LLMs (e.g., GPT-4/5, Qwen3-VL, Grok, Claude, Gemini series), web-enabled variants, and agentic deep research systems (from Google, OpenAI, Alibaba, and Perplexity). Key observations include:

• Trade-off between Writing Quality and Evidence Grounding: High-quality, fluent narrative generation does not guarantee faithful use of evidence, especially with visual content—multiple models score highly on FLAE but are penalized on TRACE (particularly VEF).
• Multimodal Capability Is Non-Monotonic: Adding vision to advanced LLMs does not always increase overall accuracy. Although multimodal models achieve better visual extraction and MOSAIC scores, they also show increased failures in fine-grained literal extraction (numerals, symbols) due to imperfect OCR and visual grounding, especially in domains demanding precise data interpretation.
• Citation and Multimodal Grounding Divergence: Reliable multimodal interpretation does not ensure reliable citation—errors frequently originate in entity disambiguation and claim-URL alignment, propagating through long-horizon evidence aggregation pipelines.
• Agentic Orchestration vs. Model Quality: The best performing DRAs, such as Gemini Deep Research, combine robust backbone LMMs with multi-step retrieval and evidence cross-verification, producing gains across all evaluation modules. Nevertheless, orchestration alone cannot compensate for weak vision-language understanding.
• System-Level Stability and Human Alignment: The multi-component evaluator exhibits high agreement (up to 73.5% pairwise, >96% Pearson on scores) with expert human reviewers. The strict VEF constraint and MOSAIC module both substantially close the gap to human preference, especially compared to vanilla LLM-as-a-judge approaches.
• Judge and Weighting Robustness: Swapping judge backbones (between Gemini 2.5 Pro and GPT-5.2) yields only modest shifts in total ranking and absolute scores, demonstrating robust pipeline design.

Implications and Future Directions

MMDeepResearch-Bench exposes persistent bottlenecks in vision-LLM grounding, fine-grained evidence extraction, and citation discipline. The benchmark’s rigorous requirement for strict visual evidence faithfulness (VEF) raises the bar for claim verification and discourages shortcutting via mere language pattern imitation. As model and agent complexity increases, maintaining entity-level and claim-URL alignment in multi-hop, multi-modal, multi-source synthesis remains a central challenge.

The open-source release of the task set, ground truths, evaluation code, and calibrated prompts will facilitate standardization in future research. Results implicate several promising directions:

• Vision-Language Fidelity: Improved OCR, chart parsing, and visual artifact disambiguation modules are needed to support robust report generation with reliable fact extraction from images.
• Agent-Environment Feedback: More sophisticated agentic search strategies that mitigate entity/reference drift across iterative retrieval and summarization steps are critical, as multi-hop reasoning amplifies small errors.
• Evaluation Stability: Continued development of task-adaptive, interpretable, and human-aligned evaluation routines is necessary for scaling benchmark coverage and tracking progress.
• Multilinguality and Generalization: Extending MMDR-Bench with fully multilingual support and additional domain coverage will further stress-test the next generation of DRAs.

Conclusion

MMDeepResearch-Bench establishes a new evaluation standard for comprehensive, citation-grounded multimodal deep research. By integrating a nuanced, fine-grained, and interpretable evaluation pipeline that aligns with expert human judgment, and by benchmarking across state-of-the-art LLMs and agents, the work lays a foundation for more trustworthy and capable multimodal research assistants. Continual improvement on the visual evidence fidelity, cross-modal alignment, and transparent evaluation criteria will be central for future progress in real-world, agentic AI research systems.

---

Reference

• “MMDeepResearch-Bench: A Benchmark for Multimodal Deep Research Agents” (2601.12346)