EchoSonar-R: A Multi-View Reasoning-Enabled Model for Disease Classification and Report Generation in Echocardiography

Abstract: Echocardiography is the most widely used non-invasive cardiac imaging modality, providing essential information for cardiovascular diagnosis. Interpreting an echocardiogram requires synthesizing complementary evidence across multiple heart views to identify abnormalities and produce structured clinical reports. While recent efforts focus on improving classification performance, most models lack explicit diagnostic reasoning and spatially grounded anatomical evidence, limiting clinician trust. We present EchoSonar-R, a multi-view reasoning-enabled vision-LLM that jointly performs multi-label disease classification and report generation from echocardiography studies. EchoSonar-R combines a spatiotemporal video encoder with a structure-aware cardiac detector that provides spatially grounded anatomical cues to improve interpretability and clinician trust during cross-view reasoning. EchoSonar-R is trained in two stages: supervised fine-tuning (SFT) on reasoning-annotated targets, followed by Group Relative Policy Optimization (GRPO) with task-specific rewards that jointly align classification and report generation within a unified reinforcement-learning framework. Across a private multi-view dataset and two public benchmarks, EchoSonar-R improves macro balanced accuracy by 17.1% on the private set and 6.1% on MIMICEchoQA over the strongest baseline, achieves a GREEN clinical faithfulness score of 0.800, and produces interpretable reasoning traces grounded in multi-view visual evidence.

• The paper introduces EchoSonar-R, which combines spatiotemporal encoding with structure-aware anatomical detection for explicit chain-of-thought reasoning in echocardiography.
• It employs a two-stage training process—supervised fine-tuning and group relative policy optimization—to enhance both multi-label disease classification and narrative report generation.
• Results demonstrate significant performance gains on private and public datasets, achieving higher clinical interpretability and reduced diagnostic errors.

EchoSonar-R: Multi-View Reasoning-Enabled Modeling for Echocardiographic Disease Classification and Report Generation

Introduction and Motivation

Echocardiography is pivotal in the diagnostic workflow for cardiovascular diseases. Expert interpretation involves integrating complementary evidence from multiple standard anatomical views to identify abnormalities and synthesize comprehensive reports. Automated approaches to echocardiographic classification and report generation have matured, yet current state-of-the-art VLMs commonly lack grounded, cross-view anatomical reasoning, limiting interpretability and clinical trust. EchoSonar-R addresses this gap by combining multi-view spatiotemporal encoding with structure-aware anatomical localization, enabling explicit chain-of-thought reasoning that integrates spatial and temporal evidence within a unified RL framework. This methodology targets robust multi-label disease classification and institution-specific narrative report generation with traceable, human-auditable rationales.

Figure 1: Schematic of the EchoSonar-R architecture, detailing the integration of frozen spatiotemporal encoders, structure-aware detectors, cross-modal projections, and two-stage training with SFT and GRPO.

Model Architecture

EchoSonar-R comprises three major components: a frozen spatiotemporal visual encoder, a structure-aware detector, and a reasoning-enabled LLM. Each video from $N$ standard views within an echocardiographic study is processed via a multiscale Vision Transformer (mViT)-based backbone, extracting dense global motion embeddings for the full cardiac cycle. In parallel, a fine-tuned RT-DETR-L structure-aware detector localizes seven key cardiac structures, with learned object query embeddings aggregated across frames to produce stable structure tokens for each anatomical region.

The resultant visual and structure representations are projected into a harmonized embedding space using two-layer MLP projectors. These are interleaved with view-identity tokens from a calibrated MobileNet-V3 view classifier, yielding a multi-view, order-preserving sequence. The LLM (LM), instantiated as Qwen3-8B, takes the concatenated sequence and executes step-wise cross-modal reasoning; outputs are serialized as $$[\langle think \rangle \mathcal{R} \langle /think \rangle \mathcal{A}]$$, where $\mathcal{R}$ denotes the reasoning trace and $\mathcal{A}$ the target answer (abnormality set or report).

Two-Stage Optimization: SFT and GRPO

Training is conducted in two distinct phases:

1. Supervised Fine-Tuning (SFT): The model is initialized with targets comprising both natural language answers and synthetic reasoning traces. Traces are automatically generated via LLMs (Qwen3-8B) anchored to clinical guidelines, ensuring coverage of both diagnostic justification and target-labeled output across a spectrum of question types. This stage optimizes for token-level likelihood, learning consistent cross-modal alignment and answer generation.
2. Group Relative Policy Optimization (GRPO): Beyond SFT, GRPO applies RL to the LLM, optimizing task-level clinical correctness via composite, differentiable reward functions. Multi-label classification rewards use IoU overlap between predicted and ground-truth abnormality sets, penalizing both omissions and hallucinated findings. Report generation rewards are computed as per-section semantic similarity, with PubMedBERT-based encodings, while report structure is directly enforced via gating rewards. A length penalty further suppresses trivial or incomplete completions. The RL stage omits the KL penalty to maximize exploration and fine decision boundary calibration.

Evaluation Protocols

EchoSonar-R was evaluated on:

• A private multi-view dataset: 1,215 test studies, 12-cardinal disease labels, and multi-section institutional reports.
• MIMICEchoQA: VQA-oriented, single-view, benchmark aligned for cross-dataset robustness.
• EchoNet-Dynamic: Large public dataset, tested on LV systolic dysfunction labels.

Classification is measured via per-disease F1, macro BAcc, sensitivity, and specificity. Reports are assessed with standard NLG metrics (BLEU, METEOR, ROUGE-L, BERTScore) and the clinically oriented GREEN score, which penalizes significant hallucinations, omissions, and anatomical location errors. Reasoning quality is auto-scored by an LLM judge (Mistral-7B), using five-point Likert scales for factuality, grounding, answer consistency, and terminology.

Numerical Results

EchoSonar-R with GRPO ($^\dagger$) achieves macro BAcc 67.4% and macro F1 49.4% on the private test set, surpassing SFT-only and all baselines by wide margins (e.g., Chiron-o1 and Lingshu at ~50% BAcc, ~24% macro F1). The improvement is robust across all conditions, and especially marked for low-prevalence abnormalities such as AV stenosis, LV enlargement, and bicuspid AV. Statistical testing confirms the improvement is non-random ($p \approx 0.006$). Performance generalizes to public sets, with a +6.1% macro BAcc increase on MIMICEchoQA relative to the strongest baseline.

Reasoning traces achieve near-boundary LLM-judged quality scores (mean 4.99/5.00), unique to EchoSonar-R and substantially above external VLMs. The model also establishes a new baseline for clinical faithfulness in report generation, with a GREEN score of 0.800 (vs. 0.491 for Lingshu and 0.358 for Chiron-o1), and achieves lowest hallucination and omission rates per report.

Figure 2: Linear probe AUROC improves for most conditions post cross-modal projection, indicating increased clinical discriminability in the learned embedding.

Component and Data Analyses

Ablations demonstrate that grounding SFT supervision with reasoning traces produces a model with higher specificity but marginally lower sensitivity—a conservative operating point favored in clinical settings. Removal of video tokens or structure tokens from the input sequence at inference degrades classification to near-random, confirming that both spatial and temporal grounding are indispensable for accurate discrimination. Cross-modal projectors significantly boost the clinical signal (mean $\Delta$AUROC +2.2), acting as task-specific adapters over the frozen encoder.

Qualitative analysis (see below) reveals EchoSonar-R can auditably synthesize evidence across all views, explicitly cross-confirm findings (e.g., LA size in both PLAX and A4C), and decompose decision logic. Baselines lack this, routinely hallucinating severe disease in normal studies or omitting key abnormal findings—an effect that carries through both reasoning and narrative outputs.

Figure 3: Structure-aware detector predictions: model localizes key cardiac structures with high anatomical fidelity in both high- and low-quality echocardiograms.

Figure 4: Detector maintains zero-shot anatomical localization on unseen public datasets, confirming the transferability of learned representations.

Implications and Future Directions

EchoSonar-R bridges the gap between vision-language modeling and trustworthy AI for echocardiography by enforcing interpretable, spatially grounded, and auditable reasoning. The integration of structure-aware detection and RL-driven optimization constitutes a strong foundation for future clinical translation of multi-modal AI systems. The architectural paradigm—combining frozen video encoders, anatomical detection heads, and reasoning-chain LM modules—may be generalized to other multi-view, high-stakes domains (e.g., multimodal radiology, serial oncology imaging).

Limitations include the lack of explicit longitudinal patient tracking—current modeling is “snapshot” based, failing to leverage priors from historical studies. Integration of longitudinal embeddings and explicit progression modeling remains a natural extension. Moreover, performance margins are largest in institutionally matched evaluation; prospective validation under domain shift and workflow integration remains necessary for clinical deployment.

Conclusion

EchoSonar-R establishes a new standard for interpretable, multi-view echocardiographic AI, incorporating anatomically explicit spatial cues and multi-step reasoning to robustly classify cardiac pathology and generate clinically accurate, traceable reports. The two-stage SFT+GRPO optimization not only maximizes numerical performance but directly aligns model behavior with clinical imperatives—minimizing hallucinations and enabling rigorous audit. These innovations present a significant advance in high-stakes vision-language modeling and offer a template for trustworthy AI development in other diagnostic imaging tasks.