Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models
Abstract: Medical image segmentation allows quantifying target structure size and shape, aiding in disease diagnosis, prognosis, surgery planning, and comprehension.Building upon recent advancements in foundation Vision-LLMs (VLMs) from natural image-text pairs, several studies have proposed adapting them to Vision-Language Segmentation Models (VLSMs) that allow using language text as an additional input to segmentation models. Introducing auxiliary information via text with human-in-the-loop prompting during inference opens up unique opportunities, such as open vocabulary segmentation and potentially more robust segmentation models against out-of-distribution data. Although transfer learning from natural to medical images has been explored for image-only segmentation models, the joint representation of vision-language in segmentation problems remains underexplored. This study introduces the first systematic study on transferring VLSMs to 2D medical images, using carefully curated $11$ datasets encompassing diverse modalities and insightful language prompts and experiments. Our findings demonstrate that although VLSMs show competitive performance compared to image-only models for segmentation after finetuning in limited medical image datasets, not all VLSMs utilize the additional information from language prompts, with image features playing a dominant role. While VLSMs exhibit enhanced performance in handling pooled datasets with diverse modalities and show potential robustness to domain shifts compared to conventional segmentation models, our results suggest that novel approaches are required to enable VLSMs to leverage the various auxiliary information available through language prompts. The code and datasets are available at https://github.com/naamiinepal/medvlsm.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
A Simple Explanation of the Paper
What is this paper about?
Doctors often need to outline (or “segment”) specific parts of medical images—like a tumor, an organ, or a lesion—to measure their size, shape, and location. This helps with diagnosis, planning surgeries, and tracking disease. The paper studies whether new AI models that understand both pictures and words (called Vision-LLMs, or VLMs) can make this segmentation easier and more flexible by using short text descriptions (called “prompts”) to guide the AI.
What questions did the researchers ask?
The paper focuses on five main questions:
- Can AI models trained on everyday photos and captions (like those from the internet) be used to segment medical images without extra training (“zero-shot”)?
- If we give these models some extra training (“finetuning”) on medical data, can they match or beat standard medical image segmentation models?
- Do text prompts actually help during training and testing, or do the models mostly rely on the image itself?
- Can these models learn from mixed datasets that include different kinds of medical images (like X-rays and ultrasound) without getting confused?
- Are these models more robust when tested on slightly different images than they were trained on (called “domain shift”)?
How did they do it?
To answer these questions, the researchers:
- Collected 11 different medical image datasets, including endoscopy (colon images), skin photos, foot ulcers, ultrasound (heart and breast), and chest X-rays.
- Built and tested four segmentation models that use both image and text:
- Two existing models trained on natural (non-medical) images: CLIPSeg and CRIS (these are Vision-Language Segmentation Models, or VLSMs).
- Two new models built from a medical VLM (BiomedCLIP), by adding a segmentation “decoder” on top. They called these BiomedCLIPSeg and BiomedCLIPSeg-D.
- Created nine types of text prompts to guide the models. Prompts could include:
- Basic class name (like “polyp” or “left ventricle”).
- Image-specific details (like size: “small,” number: “two,” location: “top-right,” shape: “round,” color: “pink”).
- General medical facts (like “benign,” patient age or gender, or whether the X-ray shows a frontal view).
- Tested the models in:
- Zero-shot mode (no extra training on medical data).
- Finetuned mode (extra training on medical images).
- Checked how changing prompt words affected performance (e.g., saying “small” instead of “large”).
- Compared these models to common segmentation methods (like UNet and DeepLabv3+).
- Evaluated performance when training on single datasets vs. pooled (combined) datasets and when testing on different but related datasets to simulate real-world differences.
To keep things simple, think of the AI as:
- A student trained on lots of everyday photos and captions.
- You can “prompt” the student with short descriptions (“Find the small round polyp in the center”), and the student tries to color in just that region of the image (segmentation).
- Finetuning is like giving the student extra lessons specifically on medical images so they perform better.
What did they find?
- Zero-shot works better on non-radiology photos:
- Without extra training, the AI models did okay on photos like skin and colon images (average scores ranged roughly from 20% to 60% Dice—Dice is a score of overlap between predicted and true regions; higher is better).
- Zero-shot did poorly on X-rays and ultrasound (these are harder, grayscale, and look quite different from everyday photos).
- Finetuning makes a big difference:
- After extra training on medical data, the models performed much better—even on X-rays and ultrasound—often reaching similar accuracy to standard segmentation models.
- Text prompts matter—but not always:
- In zero-shot mode, adding more relevant details in the prompt (like size, number, and location) often helped on endoscopy images.
- In finetuned mode, the benefit of extra prompt details was small. Just giving the class name (like “polyp”) was often enough. The models mostly relied on image features rather than text.
- One model (CRIS) learned the meaning of words like “small” or “left” very well. If you changed the prompt to the wrong description (e.g., said “large” when the polyp was small), performance dropped clearly.
- Another model (CLIPSeg) was less sensitive to word meanings during finetuning—it mostly depended on the image.
- Using pooled datasets:
- When training on combined datasets (especially multiple endoscopy datasets), the VLSMs handled the mixture well and often outperformed standard models.
- When pooling very different datasets together (like mixing endoscopy, skin, X-ray, and ultrasound), performance sometimes dropped for all models, but the VLSMs were generally more robust.
- Medical VLM vs. natural VLM:
- Surprisingly, the models built from the medical VLM (BiomedCLIP) didn’t beat the ones trained on natural images plus a segmentation decoder (CLIPSeg/CRIS).
- This suggests that being trained specifically for segmentation (on lots of segmentation data) is more important than just learning from medical texts and images in general.
Why is this important?
- Faster, more flexible tools: If AI can segment medical images guided by a short text prompt, it could help doctors quickly focus on the parts they care about, especially in interactive tools.
- Better robustness: VLSMs seem more robust to changes in data (like images from different hospitals or cameras), which is crucial in real-world settings.
- Easier to adapt: Since prompts are just text, you can describe new structures without changing the model’s architecture—handy for rare or new medical findings.
What could happen next?
- Train better “foundation” segmentation models: The study suggests that large-scale training specifically tailored to segmentation (not just image-caption matching) is key.
- Create big, high-quality “triplets”: To teach these models well, we need large datasets that have image, mask (the colored-in region), and text together.
- Improve prompt generation: Making prompts more accurate (possibly using smarter tools that read medical reports) could boost performance further.
- Go 3D: Many medical scans (CT, MRI) are 3D. Extending these ideas beyond flat 2D images would be a big step forward.
In short: AI models that understand both images and words can help segment medical images, especially when we give them some extra training. Text prompts are useful, but during training the models often lean heavily on image features. With better training and larger, well-labeled datasets, these models could make medical image analysis faster, clearer, and more reliable.
Collections
Sign up for free to add this paper to one or more collections.