MedCTA: A Benchmark for Clinical Tool Agents

Abstract: To make clinically grounded decisions, medical AI agents are expected to go beyond simple recognition and be capable of tool retrieval, evidence acquisition, and integration. Existing benchmarks largely evaluate isolated perception or single-turn question answering, and therefore provide limited visibility into failures of planning, tool recruitment, and rollout reliability. We introduce MedCTA, a benchmark for evaluating medical tool agents on clinician-validated, step-implicit tasks grounded in realistic multimodal clinical inputs, including radiology images, pathology slides, and reports. MedCTA comprises 107 real-world clinical tasks with clinician-verified executable trajectories over 5 deployed tools, and supports process-aware evaluation of tool selection, argument validity, execution stability, trajectory fidelity, and outcome quality. We benchmark 18 open- and closed-source multimodal models and find that even frontier systems remain brittle in multi-step clinical tool use: autonomous rollouts are dominated by protocol failures, premature stopping, and incorrect tool recruitment, while gold-standard tool routing yields large but still incomplete gains. These results show that strong backbone perception does not translate into reliable agentic behavior in clinical settings. MedCTA provides a rigorous testbed for auditing, diagnosing, and advancing trustworthy medical AI agents. The dataset and evaluation suite are available at https://ivul-kaust.github.io/MedCTA/

• The paper introduces MedCTA, a novel benchmark evaluating multimodal clinical tool agents with trajectory-level supervision.
• It employs a semi-automated, human-in-the-loop pipeline to curate clinician-validated tasks across diverse imaging modalities and clinical contexts.
• Results reveal significant challenges in multi-step planning and tool use accuracy, exposing reliability gaps in current LLM-based medical agents.

MedCTA: A Benchmark for Evaluating Clinical Tool Agents

Motivation and Conceptual Framework

MedCTA introduces a comprehensive benchmark centered on evaluating medical tool agents in true-to-practice, multimodal clinical workflows (2606.11702). The central motivation is to move beyond single-turn medical visual question answering (VQA) and static perception-based evaluations towards the assessment of agentic, process-aware clinical reasoning. Current medical multimodal LLM systems—while increasingly performant on static perception, closed-form classification, or QA tasks—exhibit brittle behavior when tasked with multi-step decision making requiring tool use, evidence integration, planning, and robust protocol adherence in clinical contexts. MedCTA specifically addresses the challenge of auditing these agents’ reliability by providing fine-grained, trajectory-level supervision across realistic clinical scenarios.

Dataset Design and Curation

MedCTA comprises 107 clinician-validated, step-implicit tasks grounded in authentic multimodal clinical data. Each instance is provided as a tuple consisting of the clinical context (radiology/pathology images, reports, structured data), a clinically grounded agentic query, a hidden tool subset, a clinician-verified executable trajectory, and one or more reference answers. The curation pipeline (Figure 1) employs a semi-automated, human-in-the-loop methodology, beginning with LMM-assisted drafts and finalized by domain experts to ensure queries require multi-step reasoning but do not disclose tool chains or procedural scaffolding.

Figure 1: MedCTA curation pipeline, including query construction and tool chain annotation, with dual technical and clinical validation.

Construction of reference tool trajectories proceeds through LMM-generated initial plans, strict technical annotation to enforce schema compliance and execution robustness, and clinical review to guarantee realistic, evidence-based reasoning. Figure 2 highlights the necessity and impact of this iterative annotation: human refinement substantially improves clinical soundness, eliminates over-decomposition, and aligns with medical workflows.

Figure 2: Comparison of GPT-generated vs. human-annotated trajectories, illustrating technical and clinical correction.

The benchmark is characterized by broad anatomical and modality coverage (Figure 3), spanning body-wide radiology and histopathology, and covering both imaging and textual data sources.

Figure 3: MedCTA covers varied body regions, imaging modalities, and tissue-level tasks, supporting broad clinical benchmarking.

Evaluation Metrics and Protocol

MedCTA employs a tri-level evaluation suite, measuring:

• Stepwise fidelity: Instruction adherence, tool selection precision, argument validity, and succinct summarization at each rollout step.
• Clinical reasoning: Faithfulness of logic to medical workflow, evidence integration, and semantic coverage of required findings.
• Final outcome: Goal accuracy, representing clinical task success.

Metrics are normalized to $[0,100]$, enabling direct comparison across axes. Both human clinical audits and rubric-based automated scoring ensure robust validation of reasoning steps beyond final answer correctness. Figure 4 demonstrates correlation patterns between these metrics and clinical goal accuracy, underscoring the primacy of reasoning quality over raw execution stability.

Figure 4: Pearson correlation between execution/clinical reasoning metrics and final accuracy; clinical reasoning features are strongly predictive.

Experimental Results and Analysis

Benchmarking 18 open and proprietary multimodal LLM-based agents, MedCTA exposes persistent weaknesses in autonomous controller design and multi-step tool use:

• Top-performing proprietary models (e.g., GPT-5.4, Claude-opus) reach only 31.54–31.32% outcome accuracy. The leading open model (Qwen3-8B) achieves 27.80%.
• No agent achieves non-zero strict trajectory success.
• Diagnostic answer accuracy increases by up to $+35\%$ under gold-standard routing interventions, directly exposing a controller rather than reader bottleneck.
• 64.2% of episodes fail with protocol/API errors, and under-calling (premature stopping) is pervasive at 99.2%.

This controller-centric failure profile is reinforced in Figures 10 and 12: open agents underperform primarily due to protocol errors, and performance collapses with increasing rollout horizon.

Figure 5: Open-source vs. proprietary model diagnostics, with proprietary models exhibiting higher rollout robustness under tool protocols.

Figure 6: Trajectory-length diagnostics confirm that accuracy decays rapidly with increasing tool-step sequence length, reflecting challenges in long-horizon planning.

Figure 7 indicates that even correct tool recruitment is concentrated on perception-only tasks (ImageDescription), with minimal success for more demanding tools requiring spatial localization or attribute extraction.

Figure 7: Match rate by gold tool, showing effective tool recruitment only for perception tasks and failure for region-level attribute extraction.

Fine-grained conditional analysis (Figure 8) shows that schema-conformant, API error-free rollouts are strongly associated with answer accuracy, but these are not yet achievable at scale across agent families.

Figure 8: Positive conditional advantages are concentrated in rollouts with valid schema/argument matching and zero API errors.

Failure Modes and Qualitative Trajectories

MedCTA’s diagnostic value lies in its capacity to pinpoint failure localization. Gold-standard (oracle) routing demonstrates that models possess latent clinical reasoning capabilities, but almost all autonomous failures occur before reasoning can begin due to tool misuse, protocol breakdown, or premature answer emission. Figure 9–20 and additional case studies systematically illustrate failure modes, including semantic drift, evidence-free synthesis, and horizon-induced collapse.

Figure 9: Stepwise agentic reasoning example for anatomical variant identification.

Implications and Future Directions

The results from MedCTA highlight a critical separation between backbone perception and true agentic competence. The static VQA accuracy of backbone-only VLMs is significantly higher than agentic rollout performance, indicating that reliable controller infrastructure, robust protocol adherence, and effective evidence integration remain unsolved for deployment-grade autonomous medical agents.

Practical implications include the need for targeted research in agent-controller architecture, interface protocol robustness, partial-oracle training schemes, and the integration of long-horizon planning constructs. Theoretical directions include separating reader versus controller capacity in LLM-based settings, trajectory-level intervention strategies, and the modeling of evidence accumulation and grounding under multimodal supervision.

Conclusion

MedCTA provides a rigorously constructed, clinician-verified agentic benchmark exposing actionable limitations in current LLM-based clinical tool agents. By supporting trajectory-level supervision, finely disentangling controller and reader failures, and spanning broad clinical and modality coverage, MedCTA lays the substrate for the systematic development and auditing of trustworthy, robust clinical AI agents. Its adoption is expected to accelerate progress in controller design, protocol reliability, and real-world deployment-readiness for medical assistant systems.