LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

Abstract: Conversational generative AI has demonstrated remarkable promise for empowering biomedical practitioners, but current investigations focus on unimodal text. Multimodal conversational AI has seen rapid progress by leveraging billions of image-text pairs from the public web, but such general-domain vision-LLMs still lack sophistication in understanding and conversing about biomedical images. In this paper, we propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The key idea is to leverage a large-scale, broad-coverage biomedical figure-caption dataset extracted from PubMed Central, use GPT-4 to self-instruct open-ended instruction-following data from the captions, and then fine-tune a large general-domain vision-LLM using a novel curriculum learning method. Specifically, the model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics using GPT-4 generated instruction-following data, broadly mimicking how a layperson gradually acquires biomedical knowledge. This enables us to train a Large Language and Vision Assistant for BioMedicine (LLaVA-Med) in less than 15 hours (with eight A100s). LLaVA-Med exhibits excellent multimodal conversational capability and can follow open-ended instruction to assist with inquiries about a biomedical image. On three standard biomedical visual question answering datasets, LLaVA-Med outperforms previous supervised state-of-the-art on certain metrics. To facilitate biomedical multimodal research, we will release our instruction-following data and the LLaVA-Med model.

• The paper introduces LLaVA-Med, a cost-effective biomedical multimodal assistant trained in 15 hours using a two-stage curriculum learning framework.
• It leverages biomedical image-text pairs from PMC and GPT-4 generated captions to align complex vocabulary with conversational semantics.
• Empirical evaluations show superior zero-shot performance on biomedical VQA datasets, highlighting its potential in healthcare diagnostics.

Analyzing LLaVA-Med's Approach to Robust Biomedical Multimodal Assistants

The paper presents LLaVA-Med, a significant effort to develop a multimodal language-and-vision model specifically tailored for biomedical applications, leveraging recent advances in large vision-language (VL) models and generative AI. It emphasizes constructing a cost-effective framework for training such a sophisticated assistant within a mere 15-hour period using high-performance computational resources (eight A100 GPUs).

Methodological Approach

LLaVA-Med is built upon a curriculum learning framework that emphasizes two distinct training stages. The first stage involves aligning complex biomedical vocabulary by using a large corpus of biomedical image-text pairs from PubMed Central (specifically the PMC-15M dataset). In the second stage, the model is refined to follow open-ended conversational semantics with data generated by GPT-4 from the figure captions. This curriculum mimics the progressive acquisition of specialized knowledge, distinguishing laypersons from experts in the domain.

Crucially, the paper exploits self-instruct capabilities of GPT-4 to create training datasets without manual annotations, a move that substantially enhances the breadth and diversity of the model's multimodal instruction-following capabilities. This methodology not only underscores the efficiency of the training process but also leverages the scale and specificity of the PMC-15M dataset to provide a richer contextual training ground.

Empirical Results and Evaluation

The empirical results showcase LLaVA-Med's superior performance across standard biomedical VQA datasets, notably surpassing previous state-of-the-art models on several metrics. Noteworthy is the model's excellence in zero-shot scenarios for datasets like VQA-RAD and PathVQA. The model's ability to transpose a layperson's conceptual alignment into a domain-specific expert understanding affirms the success of its training approach.

Evaluations employing an expert evaluation tool like GPT-4 for scoring provide a strategic look at the model's understanding capabilities, which are essential metrics for conversational agents. Such a comprehensive evaluation highlights LLaVA-Med's adeptness at navigating complex biomedical dialogues and tasks, underpinning its potential usefulness in real-world healthcare environments.

Implications and Future Prospects

The paper sets a vital precedent in developing domain-specific assistants by demonstrating that large language and vision models can be fine-tuned to cater to different specialized fields effectively and cost-effectively. The release of LLaVA-Med’s datasets and training code also speaks to its potential in fostering further research and development in biomedical multimodal systems.

As AI continues to permeate various professional domains, LLaVA-Med gives us a glimpse of future applications where domain-specific AIs substantiate their roles in healthcare, reducing workload, enhancing decision support systems, and potentially improving patient outcomes through better diagnostics and research interpretations. Future iterations of models like LLaVA-Med might integrate more sophisticated knowledge graphs and reasoning capabilities, paving the way for multimodal assistants that not only respond accurately but reason and suggest actionable insights efficiently.

Conclusion

In conclusion, LLaVA-Med represents an essential step towards creating viable, domain-focused multimodal language-and-vision models. It successfully bridges the gap between general-purpose models and specialized needs in the biomedical field. The research underscores the efficacy of using structured datasets and powerful LLMs to achieve superior zero-shot performance and sets the foundation for further innovations in domain-specific AI assistants. Such advancements will undoubtedly contribute significantly to the evolution of AI's role in highly specialized industries like healthcare.