EchoPrime: A Multi-Video View-Informed Vision-Language Model for Comprehensive Echocardiography Interpretation

Abstract: Echocardiography is the most widely used cardiac imaging modality, capturing ultrasound video data to assess cardiac structure and function. AI in echocardiography has the potential to streamline manual tasks and improve reproducibility and precision. However, most echocardiography AI models are single-view, single-task systems that do not synthesize complementary information from multiple views captured during a full exam, and thus lead to limited performance and scope of applications. To address this problem, we introduce EchoPrime, a multi-view, view-informed, video-based vision-language foundation model trained on over 12 million video-report pairs. EchoPrime uses contrastive learning to train a unified embedding model for all standard views in a comprehensive echocardiogram study with representation of both rare and common diseases and diagnoses. EchoPrime then utilizes view-classification and a view-informed anatomic attention model to weight video-specific interpretations that accurately maps the relationship between echocardiographic views and anatomical structures. With retrieval-augmented interpretation, EchoPrime integrates information from all echocardiogram videos in a comprehensive study and performs holistic comprehensive clinical echocardiography interpretation. In datasets from two independent healthcare systems, EchoPrime achieves state-of-the art performance on 23 diverse benchmarks of cardiac form and function, surpassing the performance of both task-specific approaches and prior foundation models. Following rigorous clinical evaluation, EchoPrime can assist physicians in the automated preliminary assessment of comprehensive echocardiography.

• The paper introduces EchoPrime, a multi-view vision-language model that integrates echocardiogram videos and text reports using contrastive learning on over 12 million video-report pairs.
• It achieves superior performance with a mean AUC of 0.92 across 17 tasks and low mean absolute errors in assessing left ventricular systolic function.
• The model's architecture, featuring view classification and anatomical attention, enhances cross-modal retrieval and supports robust disease prediction.

Analysis of EchoPrime: A Vision-LLM for Echocardiography Interpretation

EchoPrime is a significant advancement in the use of vision-LLMs within the domain of echocardiography. This paper introduces a multi-view, view-informed video-based model designed to synthesize multi-modal data and enhance the clinical interpretation of echocardiograms by integrating video and text information. This work addresses the limitations of single-view, single-task models by training on an extensive dataset of over 12 million video-report pairs, achieving state-of-the-art performance across various benchmarks.

Methodology and Model Architecture

EchoPrime utilizes a contrastive learning approach to train unified embeddings for echocardiography videos alongside their respective text reports. The model is composed of several components:

• Video Encoder and Text Encoder: These encoders map video content and textual data into a shared latent space, allowing for holistic analyses of echocardiographic studies.
• View Classifier and Anatomical Attention Module: These components are pivotal for weighting the importance of each view based on anatomical relevance, facilitating comprehensive interpretation tasks through multi-instance learning.

This multi-modal design is integral to EchoPrime's capacity for synthesizing complex information from numerous views, aligning with practices utilized by cardiologists.

Performance Evaluation

EchoPrime's performance is rigorously evaluated against previous foundation models and task-specific AI systems across a wide array of benchmarks. Notably, EchoPrime outperformed existing models such as BioMedCLIP and EchoCLIP with a mean AUC of 0.92 on 17 classification tasks. Significant results include an improvement in predicting left ventricular systolic function, with a mean absolute error (MAE) of 4.8% on the internal dataset and 4.1% on an external dataset.

Cross-Modal Retrieval and Interpretability

The model's architecture significantly enhances its retrieval capabilities. EchoPrime achieved a Recall@10 of 98% for video-to-text tasks, demonstrating superior retrieval accuracy compared to EchoCLIP. The integration of multi-view data and anatomical attention facilitates an interpretable framework, balancing assessments from various anatomical perspectives in alignment with expert cardiologists.

Transfer Learning and Disease Prediction

EchoPrime also exhibits potential for transfer learning, demonstrating superior accuracy in both echocardiographic and broader medical diagnosis tasks. The model's embedding space proved effective in identifying cardiac diseases not typically diagnosed via echocardiography, such as myocardial infarction and cardiac amyloidosis, achieving impressively high AUC scores.

Implications and Future Work

The implications of EchoPrime are substantial for both clinical practice and AI research. By providing a robust, multi-task framework, this model can automate preliminary echocardiogram assessments and support clinician workflows. Future research directions should focus on applying EchoPrime across more diverse clinical settings and integrating other diagnostic modalities to enhance utility. Additionally, the potential for AI deployment in lower-resource environments presents intriguing opportunities for expanding healthcare access.

EchoPrime is a noteworthy contribution to medical AI, integrating comprehensive echocardiographic data for improved diagnostic accuracy. Its establishment of effective multi-modal processing sets a precedent for further research and application within the field.