DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning

Abstract: Specialized visual tools can augment LLMs or vision LLMs with expert knowledge (e.g., grounding, spatial reasoning, medical knowledge, etc.), but knowing which tools to call (and when to call them) can be challenging. We introduce DART, a multi-agent framework that uses disagreements between multiple debating visual agents to identify useful visual tools (e.g., object detection, OCR, spatial reasoning, etc.) that can resolve inter-agent disagreement. These tools allow for fruitful multi-agent discussion by introducing new information, and by providing tool-aligned agreement scores that highlight agents in agreement with expert tools, thereby facilitating discussion. We utilize an aggregator agent to select the best answer by providing the agent outputs and tool information. We test DART on four diverse benchmarks and show that our approach improves over multi-agent debate as well as over single agent tool-calling frameworks, beating the next-strongest baseline (multi-agent debate with a judge model) by 3.4% and 2.4% on A-OKVQA and MMMU respectively. We also find that DART adapts well to new tools in applied domains, with a 1.3% improvement on the M3D medical dataset over other strong tool-calling, single agent, and multi-agent baselines. Additionally, we measure text overlap across rounds to highlight the rich discussion in DART compared to existing multi-agent methods. Finally, we study the tool call distribution, finding that diverse tools are reliably used to help resolve disagreement.

• The paper introduces DART, a framework that detects agent disagreements and recruits expert tools to resolve contentious interpretations.
• It employs a multi-stage pipeline integrating VLM reasoning with targeted tool intervention, achieving up to 68.9% accuracy on key benchmarks.
• Experimental results highlight DART's modular design, outpacing traditional debate methods by using disagreement as a signal for dynamic tool activation.

DART: Leveraging Multi-Agent Disagreement for Tool Recruitment in Multimodal Reasoning

Introduction and Motivation

DART proposes a multi-agent framework for visual question answering (VQA) that dynamically leverages external vision tools based on the observed disagreements among multiple vision-LLM (VLM) agents. The work addresses the empirical limitation of both unimodal and multimodal multi-agent debate frameworks, in which collaborative discussion often stagnates due to agents sharing overlapping perceptual failures and lacking external sources of ground truth to resolve contentious points. DART draws on the finding that VLMs often underperform specialized expert models (e.g., for grounding, OCR, spatial reasoning) and accordingly recruits such independent tools adaptively during agent debate to inject new and more reliable information where it is most needed.

The key insight is that productive debate requires targeted introduction of novel evidence, precisely where agents diverge, instead of static or “top-down” tool calling solely guided by the initial query. This targeted recruitment, grounded in explicit model disagreement, yields richer dialog, improved answer calibration, and higher overall accuracy across challenging multimodal tasks (Figure 1).

Figure 1: Schematic comparison of (A) multi-agent debate, (B) top-down LLM tool-calling, and (C) the DART approach, which injects expert tool interventions conditioned on detected agent disagreements.

Methodology

Framework Overview

DART’s pipeline consists of five consecutive stages aimed at systematically improving answer reliability and robustness:

1. Initial Answer Generation: Multiple VLM agents generate independent answers, detailed reasoning chains, and confidence estimates for a given (image, question) tuple.
2. Tool-Based Disagreement Resolution: An LLM-based recruitment agent compares the agents’ outputs, identifies points of nontrivial disagreement, and selects an optimal set of expert tools (object detectors, grounders, OCR, etc.) to resolve each specific contention.
3. Tool-Based Agreement Scoring: Each agent’s output is compared against the expert tools’ outputs using an LLM agreement scorer, yielding per-tool alignment scores. These are averaged to assign each agent an expert-aligned trust score.
4. Discussion: Agents receive all prior answers, tool outputs, and alignment scores, then revise their responses and reasoning accordingly. They are explicitly instructed to defer to expert tool outputs in cases of contradiction.
5. Aggregation: A dedicated aggregator VLM reviews all agent responses (with scores and tool information) and produces the final answer via chain-of-thought reasoning.

Immediately after this methodological description, visual guidance establishes a concrete understanding of how information flows and is augmented through each step.

Figure 2: DART’s pipeline—beginning with agent answer generation, proceeding through targeted tool intercession and alignment scoring, and culminating in collaborative revised discussion and aggregation.

Expert Tool Pool

The pool of domain tools includes both specialized models (e.g., GroundingDINO for grounding, YOLOv11 for object detection, SpaceLLaVA for spatial reasoning, OCR-specific models, and MedGemma for medical imaging analysis) and strong VLMs prompted for specific subtasks (e.g., captioning, attribute extraction, meta-reasoning). This hybrid approach allows complementary perceptual and semantic signals throughout the system.

DART is agnostic to the concrete set of agents and tools, thus permitting easy expansion to specialty domains by adding new expert modules without retuning the debate or aggregation protocols.

Disagreement-Driven Tool Use

Distinct from prior top-down agent+tool paradigms (e.g., ViperGPT, Chameleon), DART triggers tool invocation dynamically: only if and where explicit agent-agent disagreement is detected, and with each tool’s invocation justified directly by the nature of the disagreement. This yields efficient use of resources and narrows the focus to the queries’ information bottlenecks.

Experimental Results

DART is evaluated on four representative VQA-style benchmarks: A-OKVQA (commonsense/world knowledge), MMMU (multi-discipline), NaturalBench (adversarial VQA), and M3D-VQA (medical images). All settings operate under a zero-shot or training-free regime to emphasize generalizability and the value of tool-based augmentation over fine-tuned single-model baselines.

Figure 3: DART’s tool usage distribution on A-OKVQA, demonstrating balanced deployment of multiple expert tools during real-world VQA.

Performance Comparison

DART consistently outperforms both single-agent and multi-agent debate baselines, as well as training-free tool-calling systems:

• On A-OKVQA, DART achieves 68.9% accuracy, surpassing the best multi-agent debate baseline by 3.4% and the best single-agent (Self-Consistency/Ovis2) by 4.3%.
• On MMMU and NaturalBench, analogous gains are observed, with improvements of 2.4% and 1.2%, respectively, over competitive multi-agent debate.
• Against ViperGPT and Chameleon, DART shows substantial margins (>10%) on vision-heavy tasks, validating its disagreement-driven tool selection.

These results reflect DART’s ability to catalyze meaningful agent discussion by injecting targeted, high-signal expert knowledge precisely at bottleneck points.

Figure 4: Comparative improvement of DART and multi-agent debate across three rounds; error bars show inter-agent variance. The largest gains occur after the initial expert tool intervention, with diminishing marginal improvements in subsequent rounds.

Domain Adaptability

DART’s modular design enables seamless incorporation of new expert tools for specialized domains. For M3D-VQA (medical VQA), introduction of MedGemma as a medical expert resulted in a further 1.5% boost over all baselines, including both single-agent and multi-agent methods, highlighting adaptability to settings where generalist VLMs alone are inadequate.

Discussion Quality and Diversity

Textual overlap analysis (e.g., ROUGE-L, Jaccard index) demonstrates that DART discussions exhibit significantly lower redundancy compared to pure multi-agent debate, indicating richer, more divergent reasoning in successive rounds. The explicit inclusion of tool outputs and expert scores introduces authentic novel evidence into the agent conversations, rather than reinforcing preexisting priors.

Ablation Studies

Systematic ablations reveal:

• Removal of diverse answering agents or vision-expert modules each yields ≥3% accuracy drop, confirming the importance of both diversity and external evidence.
• Substitution of all specialist tools with a single VLM leads to a 2.9% performance reduction, demonstrating that generalized VLMs are insufficient to replace fine-tuned expert tools for certain perceptual subproblems.
• Omitting agreement scoring produces ≈1% drop, establishing that score-based calibration is additive but less impactful than diverse agents and direct tool use.

Qualitative Examples

DART’s operation is further illustrated through real-world VQA scenarios where standard agents fail but targeted tool calls enable decisive resolution.

Figure 5: Example—Initial agent disagreement on parking meter enforcement; OCR tool retrieval resolves the disagreement, guiding the agents to converge on the gold answer.

Figure 6: Example—Conflicting agent perceptions of the direction of a flag; spatial and captioning tools clarify, enabling final correct aggregation.

Implications and Future Directions

From a practical perspective, DART’s disagreement-driven tool recruitment paradigm outlines a scalable avenue for reliable multimodal reasoning without heavy retraining or rigid orchestration templates. By systematically leveraging tool diversity and agent disagreement, DART supports higher factual robustness and can be extended to any domain with appropriate external models or APIs. The results suggest a path forward for open-ended, collaborative, and tool-augmented AI systems that are robust to the limitations of any single model.

Theoretically, the framework elevates the role of disagreement as a first-class signal for active knowledge solicitation, distinct from passive ensembling or (over-)reliance on self-reported confidence. Future extensions could include further integration of planning agents, continual tool/agent pool optimization, or explicit modeling of debate-theoretic strategies to maximize information gain per tool-call.

Conclusion

DART introduces a principled and empirically validated framework for enhancing multimodal reasoning via multi-agent disagreement detection and dynamic expert tool recruitment. Experimental evidence across four benchmarks demonstrates superior performance, adaptability, and richer inter-agent communication compared to prior single- and multi-agent models and static tool-calling paradigms. The system effectively combines the generalist reasoning of VLMs with domain-specific perceptual strengths of independent tools, setting a foundation for future systems that coordinate diverse AI modules at run time in open environments.