Echo-α: Large Agentic Multimodal Reasoning Model for Ultrasound Interpretation

Abstract: Ultrasound interpretation requires both precise lesion localization and holistic clinical reasoning, yet existing methods typically excel at only one of these capabilities: specialized detectors offer strong localization but limited reasoning, whereas multimodal LLMs (MLLMs) provide flexible reasoning but weak grounding in specialized medical domains. We present Echo-α, an agentic multimodal reasoning model for ultrasound interpretation that unifies these strengths within an invoke-and-reason framework. Echo-α is trained to coordinate organ-specific detector outputs, integrate them with global visual context, and convert the resulting evidence into grounded diagnostic decisions beyond detector-only inference. This behavior is established through a nine-task supervised curriculum and then refined by sequential reinforcement learning under different reward trade-offs, yielding Echo-α-Grounding for lesion anchoring and Echo-α-Diagnosis for final diagnosis. On multi-center renal and breast ultrasound benchmarks, Echo-α outperforms competitive baselines on both grounding and diagnosis. In particular, on cross-center test sets, Echo-α-Grounding attains 56.73%/43.78% [email protected] and Echo- α-Diagnosis reaches 74.90%/49.20% overall accuracy on renal/breast ultrasound. These results suggest that agentic multimodal reasoning can turn specialized detectors into verifiable clinical evidence, offering a practical route toward ultrasound AI systems that are more accurate, interpretable, and transferable. The repository is at https://github.com/MiliLab/Echo-Alpha.

• The paper introduces an agentic framework that integrates MLLM reasoning with specialized ultrasound lesion detectors via an invoke-and-reason loop.
• The model employs supervised curriculum fine-tuning and reinforcement learning to optimize spatial lesion localization and diagnostic accuracy across multiple datasets.
• The results reveal consistent performance gains and adaptability across detectors and centers, underscoring its potential for interpretable clinical AI.

Echo-α: An Agentic Multimodal Reasoning Model for Ultrasound Interpretation

Motivation and Background

Diagnostic ultrasound interpretation demands both robust lesion localization and sophisticated clinical reasoning. Traditional automatic approaches—specialized detectors and generic Multimodal LLMs (MLLMs)—are fundamentally limited. Detectors offer reliable spatial grounding but poor interpretability and holistic assessment; MLLMs excel at contextual reasoning but exhibit unreliable spatial precision and hallucination, particularly within medical domains. The Echo-α framework proposes an agentic solution: orchestrating MLLM-based reasoning with domain-specific detection tools and employing an invoke-and-reason loop for verifiable clinical evidence generation.

Figure 1: Echo-α’s invoke-and-reason loop integrates internal visual reasoning with organ-specific detector calls, covering spatial grounding, diagnostic reasoning, report generation, and tool collaboration, with RL driving alignment for localization quality, correctness, geometric consistency, and tool efficiency.

Methodology

Echo-α is instantiated atop a pretrained MLLM backbone, acting as a diagnostic agent that dynamically interacts with domain-specific, callable lesion detectors (e.g., LW-DETR variants for renal/breast ultrasound). The workflow begins with visual hypothesis formation, leverages function calls to detectors for candidate boxes and predictions, interprets detector outputs (visual overlays and structured metadata), and synthesizes grounded clinical decisions.

An explicit multi-stage training procedure includes:

• Supervised Curriculum Fine-Tuning (SFT): A nine-task curriculum spans referring expression comprehension/generation, direct and attribute-based diagnosis, multi-step analysis, detector collaboration (box refinement/class correction), and agentic tool invocation loop skills. Training targets are derived from teacher model rationales via teacher forcing.
• Reinforcement Learning (RL): Sequential RL employing Group Relative Policy Optimization (GRPO) optimizes the model for distinct objectives: lesion anchoring (localization, shape) vs. final diagnosis (classification accuracy, contextual reasoning). Reward structure includes smooth IoU-based localization, class accuracy contingent on spatial precision, Distance-IoU shape alignment, and strategic tool usage costs.

Experimental Results

Echo-α is validated on multi-center datasets for renal and breast ultrasound interpretation (COCO-format and BI-RADS annotation). Evaluation follows same-center validation and cross-center testing protocols, emphasizing transferability. The model is compared against several baselines: specialized detectors, backbone MLLM (Qwen3-VL) direct prompting, tool-augmented MLLM, SFT-only, SFT with tool, and RL-optimized variants.

Key findings include:

• Grounding: Echo-α achieves top [email protected] performance across all splits, e.g., 70.78% (renal Val), 56.73% (renal Test), 50.37% (breast Val), and 43.78% (breast Test), surpassing Direct MLLM + tool by up to +14.22 (Val) and +6.62 (Test) points.
• Diagnosis: Echo-α attains superior accuracy—77.43% (renal Val), 74.90% (renal Test), 48.75% (breast Val), 49.20% (breast Test)—with consistent gains over Direct MLLM + tool, up to +13.44 (Val), +7.91 (Test).
• Transferability: Performance improvements persist across cross-center conditions, indicating the agentic framework generalizes detector reasoning beyond single-center distribution fitting.

Detector-Agnostic Tool Generalization

Echo-α is robust to varied detector architectures (YOLO-series, LW-DETR-DINOv3, RF-DETR-DINOv3). The agent consistently enhances diagnosis accuracy independent of detector backbone, with gains ranging from +2.15 to +22.95 points. Improvement magnitude is inversely proportional to detector strength; sparse/low-performing detectors benefit from contextual evidence completion, dense/noisy detectors from candidate verification and suppression.

Qualitative Case Studies

Case analysis reveals Echo-α’s capacity to align candidate evidence with holistic diagnostic decision-making:

Figure 2: In renal and breast ultrasound cases, Echo-α improves spatial lesion alignment and refines diagnosis by combining detector feedback, lesion appearance, context, and clinical cues, distinguishing among conflicting interpretations.

For renal ultrasound, Echo-α translates detector-grounded evidence (heterogeneous echogenicity, indistinct margins, structural distortion) into a malignant tumor diagnosis, correcting the benign preference of the baseline. In breast ultrasound, Echo-α integrates multi-candidate BI-RADS assessments and clinical data, resolving ambiguity and yielding contextually correct predictions.

Implications and Future Directions

Echo-α demonstrates that agentic multimodal reasoning frameworks can transform detector modules from static predictors to revisable, verifiable clinical evidence, with enhanced interpretability and accuracy. The invoke-and-reason paradigm—reinforcing explicit tool interaction, spatial grounding, and contextual synthesis—provides a blueprint for extensible and transferrable medical AI systems.

Practically, Echo-α offers a route to explainable, agentic ultrasound interpretation, with favorable generalization across centers and detector architectures. Theoretically, the results underscore the utility of sequential reinforcement learning for optimizing competing objectives within medical VQA, suggesting broader applicability to other imaging domains.

Prospective advances may involve integrating additional clinical modalities (e.g., textual EMR history), scaling to more anatomical regions, and refining RL reward schemes to balance interpretability and clinical plausibility. Extension to real-time, interactive diagnostic assistance and cross-modal agentic workflows constitutes a logical progression.

Conclusion

Echo-α unifies detector precision and MLLM reasoning in agentic ultrasound interpretation via an invoke-and-reason loop, supervised curriculum, and reward-driven optimization. The model consistently outperforms both detector-only and generic MLLM baselines in grounding and diagnosis, exhibits strong transferability, and adapts effectively across varied detector architectures. The research validates agentic multimodal reasoning as a foundation for accurate, interpretable, and extensible ultrasound AI systems, with broad implications for clinical imaging AI.