Unified Ultrasound Intelligence Toward an End-to-End Agentic System

Abstract: Clinical ultrasound analysis demands models that generalize across heterogeneous organs, views, and devices, while supporting interpretable workflow-level analysis. Existing methods often rely on task-wise adaptation, and joint learning may be unstable due to cross-task interference, making it hard to deliver workflow-level outputs in practice. To address these challenges, we present USTri, a tri-stage ultrasound intelligence pipeline for unified multi-organ, multi-task analysis. Stage I trains a universal generalist USGen on different domains to learn broad, transferable priors that are robust to device and protocol variability. To better handle domain shifts and reach task-aligned performance while preserving ultrasound shared knowledge, Stage II builds USpec by keeping USGen frozen and finetuning dataset-specific heads. Stage III introduces USAgent, which mimics clinician workflows by orchestrating USpec specialists for multi-step inference and deterministic structured reports. On the FMC_UIA validation set, our model achieves the best overall performance across 4 task types and 27 datasets, outperforming state-of-the-art methods. Moreover, qualitative results show that USAgent produces clinically structured reports with high accuracy and interpretability. Our study suggests a scalable path to ultrasound intelligence that generalizes across heterogeneous ultrasound tasks and supports consistent end-to-end clinical workflows. The code is publicly available at: https://github.com/MacDunno/USTri.

• The paper introduces a tri-stage pipeline (USGen, USpec, and USAgent) that unifies multi-task ultrasound analysis with deterministic outputs.
• It employs a hybrid TransUNet-ViT encoder with lightweight, task-specific adapters to overcome domain shifts and optimize multi-task performance.
• Experimental results on the FMC_UIA benchmark, including DSC of 0.8980 and improved metrics, validate its clinical scalability and interpretability.

USTri: A Tri-Stage Unified Agentic Pipeline for Ultrasound Intelligence

Motivation and Context

The diversity of clinical ultrasound data, spanning heterogeneous organs, anatomical views, acquisition devices, and task objectives, poses major barriers to scalable, workflow-level deployment of learning-based systems. Most existing approaches use either task-specific models or unstable, monolithic multi-task learning, resulting in poor domain generalization and lack of interpretability at the workflow level. Prior foundation models—such as USFM—demonstrate improved transferability for isolated downstream tasks but do not deliver robust, unified, end-to-end pipelines capable of practical multi-organ and multi-task operation with deterministic, interpretable clinical outputs.

Methodology: The USTri Tri-Stage Pipeline

USTri introduces a comprehensive tri-stage pipeline—USGen, USpec, and USAgent—explicitly designed to synergize unified representation learning, robust task specialization, and agentic workflow composition.

Figure 1: Overview of the proposed USTri pipeline, highlighting USGen, USpec, and USAgent in a unified end-to-end structure for heterogeneous ultrasound analysis.

Stage I (USGen): Universal Generalist Modeling

USGen is trained as a universal backbone over a diverse cohort of organs, anatomical views, and supervision types (segmentation, classification, detection, regression). It employs a TransUNet-style hybrid encoder, leveraging both ViT token embeddings and high-resolution convolutional stem features to encode domain-general cues robust to acquisition variability and operator-dependence. Multi-task learning is stabilized by a dataset-rotating schedule that iteratively optimizes per-head losses, avoiding cross-task interference.

Stage II (USpec): Parameter-Efficient Specialization

Stage II freezes the USGen backbone and efficiently finetunes lightweight, dataset-specific heads (adapters for classification, bespoke heads for other tasks). This achieves dataset-level task alignment and robustness to view/device/annotation shifts. The approach improves per-domain accuracy and mitigates the compromise between generalization and specialization characteristic of joint end-to-end training.

Stage III (USAgent): Workflow-Level Agentic Inference

USAgent orchestrates the ensemble of USpec heads as callable tools within a biomedical VLM-planner (LLaVA-Med v1.5). The VLM planner composes multi-step inference chains (e.g., Detect→Segment→Regress or Detect→Segment→Classify) by querying appropriate heads with structured parameters. USAgent maintains a lightweight state cache for intermediate results, guaranteeing deterministic, auditable, and interpretable report generation instead of free-form outputs. Termination occurs via planner-issued null actions.

Experimental Results and Quantitative Analysis

Evaluation is conducted on the FMC_UIA Challenge benchmark, a multi-center, cross-domain ultrasound dataset comprising 27 tasks spanning four task types under stringent domain shift (unseen validation domains). Metrics include Dice, HD for segmentation, AUC/F1/MCC for classification, IoU for detection, and MRE for regression—computed at clinical resolution.

Key numerical findings highlight several claims:

• USpec (Stage II) achieves superior performance on every metric versus all baselines (including USFM and official multi-head MTL), e.g., DSC: 0.8980, HD: 27.21, IoU: 0.8000, MRE: 18.42.
• Full supervision with task-specific alignment outperforms generic self-supervised pretraining, especially on geometry-sensitive metrics (HD, IoU, MRE), demonstrating the importance of explicit, label-driven optimization over broad, label-efficient features.
• Two-stage learning (USGen→USpec) yields consistent, robust gains versus end-to-end or pure pretrain-finetune paradigms, reducing overfitting and enhancing adaptation to annotation and device shift.

Qualitative Insights and Workflow Demonstrations

USAgent's ability to generate clinical workflow-style outputs is illustrated in labor progress assessment and breast lesion diagnosis.

Figure 2: USAgent realizing (a) Detect-Segment-Regress for intrapartum ultrasound, and (b) Detect-Segment-Classify for breast lesion diagnosis, with structured, evidence-backed reporting.

In both scenarios, USAgent automates complex multi-step clinical procedures with transparent evidence trails (ROI masks, segmentation contours, quantitative measurements), directly supporting downstream interpretability and clinical decision support. The structure and auditability of these outputs contrast favorably with end-to-end black box or free-form LLM-based reporting.

Theoretical and Practical Implications

USTri demonstrates that robust, scalable ultrasound intelligence is achievable by integrating foundation modeling with lightweight specialization and workflow-level agentic inference—all within a deterministic, auditable pipeline. This agentic paradigm provides a modular, extensible template: universal representation learning mitigates data heterogeneity, parameter-efficient adapters address domain/task idiosyncrasies, and VLM-driven tool composition supports complex, clinically relevant workflows with deterministic outputs.

Practically, these findings indicate a route towards clinical deployment of AI systems capable of seamlessly handling real-world multi-organ, multi-task variation, with embedded interpretability, auditability, and workflow conformity.

Theoretically, USTri corroborates the relevance of explicit supervision, modular specialization, and restricted agentic composition for robust generalization under domain shift—challenging the sufficiency of broad self-supervised pretraining or monolithic foundation models in safety-critical clinical pipelines.

Future Perspectives

Potential extensions include incorporating continual learning for real-time adaptation to emergent domains, integrating non-image modalities (e.g., reports, waveforms), and refining VLM planners toward explanation-aware decision support. The USTri agentic framework offers a template for other domains (beyond ultrasound) where workflow granularity, heterogeneous task composition, and robust, interpretable automation are required.

Conclusion

USTri establishes a tri-stage agentic pipeline that unifies generalist modeling, robust specialization, and clinically structured, interpretable workflow composition for ultrasound intelligence. Achieving state-of-the-art performance on all tasks and offering transparent agentic reporting, this work demonstrates an effective path toward scalable, deployable, and trustworthy multi-task clinical AI systems (2604.16914).