Medical SAM3: Universal Medical Segmentation

Updated 21 January 2026

Medical SAM3 is a universal segmentation model that applies prompt-driven techniques (text, points, boxes) to accurately segment diverse medical images.
The model employs full end-to-end adaptation of the SAM3 architecture, achieving mean Dice scores up to 77% and robust generalization across unseen datasets.
By leveraging comprehensive semantic supervision and multi-modality training, Medical SAM3 handles severe domain shifts and complex anatomical details effectively.

Medical SAM3 refers to the adaptation of the third-generation Segment Anything Model (SAM3)—originally developed as a large-scale universal segmentation foundation model—to the complex and heterogeneous domain of medical image segmentation. Medical SAM3 targets universal, prompt-driven segmentation of both 2D and 3D medical images and videos across diverse imaging modalities (e.g., radiography, CT, MRI, ultrasound, pathology, endoscopy). The core innovation is an architectural and training pipeline modification that holistically adapts the SAM3 backbone, including the vision, prompt, and text encoders as well as the mask decoder, to robustly handle severe domain shift and semantic ambiguity in medical image data (Jiang et al., 15 Jan 2026). The model achieves consistent, high performance under both spatial (e.g., point, box) and open-vocabulary textual prompts, establishing itself as a universal foundation model for medical segmentation.

1. Model Architecture and Adaptation Paradigm

Medical SAM3 builds directly on the modular architecture of SAM3, which integrates: a ViT-based vision encoder producing patchwise spatial representations; a text encoder to embed open-vocabulary concept prompts; a geometric prompt encoder (points/boxes); and a shared Transformer-based mask decoder supporting detection and tracking for video/volumetric applications.

Distinctive adaptations include:

Full-parameter adaptation: All backbone components (vision, text, prompt encoders, mask decoder) are fine-tuned end-to-end on medical data, with no new layers or adapter modules. Layer-wise learning rate decay (LLRD) specializes deeper layers while retaining generic low-level filters.
High-resolution processing: Inputs are standardized to 1008 × 1008 pixels for 2D slices, capturing fine anatomical details and supporting precise regional segmentation.
Unified 2D representation for 3D/4D data: Volumetric scans are mapped to ordered 2D frames; a detector–tracker cascade (inherited from the SAM3 video mode) enables semantic instance detection and cross-slice mask propagation.
Prompt flexibility: Text-only, geometric (points, boxes), or hybrid concept prompts are supported, enabling both open-vocabulary and spatially guided segmentation.

The mask decoding process executes cross-attention between spatial, textual, and geometric cues and generates instance-level mask predictions for each prompt.

2. Fine-Tuning Strategy and Loss Functions

The adaptation pipeline for Medical SAM3 is characterized by comprehensive, instance-level supervision and a multi-term objective:

Training regime: The model is fine-tuned with AdamW, group-wise learning rates, linear warmup, and subsequent inverse-square-root decay. LLRD with γ = 0.85 is applied across the ViT backbone specifically for medical domain adaptation.
Set-prediction loss: Following DETR-style training, mask queries and ground-truth masks are assigned one-to-one by Hungarian matching. The loss is the sum of matched finding loss (classification, presence, localization terms) and per-pixel segmentation loss (focal + Dice + segmentation presence).
Purely semantic supervision: During training, only text prompts describing the target concept (e.g., “optic nerve,” “colonic tumor”) are used, without privileged spatial hints. This compels the model to learn direct semantic-to-spatial alignment.
No architectural modifications: The approach strictly adapts weights, demonstrating that holistic fine-tuning suffices to overcome the substantial semantic gap between natural and medical imaging (Jiang et al., 15 Jan 2026).

3. Multi-Modality Dataset Curation and Preprocessing

Medical SAM3 is trained to universality via aggregation of 33 public datasets spanning 10 image modalities and 263,705 mask annotations. These include:

Radiography and angiography (CXR, BTXRD, ARCADE, etc.)
CT and MRI (multiple anatomical regions and tasks)
Ultrasound, endoscopy, fundus/OCT, pathology, histopathology, microscopy, dermoscopy
2D and 3D/4D data, with volumetric images sliced and processed in frame-wise order

Preprocessing consists of intensity normalization, center cropping/padding, and mapping all slices to a uniform spatial resolution (1008 × 1008). Text prompts are standardized as short, canonical clinical concept phrases. During training, an 85/15 train/validation split is enforced per dataset.

4. Quantitative and Qualitative Performance

Medical SAM3 demonstrates consistent and substantial improvements in prompt-driven segmentation under domain shift:

Internal benchmarks: On 10 held-out sets, Medical SAM3 achieves mean Dice 77.0% (IoU 67.3%), compared to baseline SAM3 (Dice 54.0%, IoU 43.3%) under text-only prompting protocols.
External zero-shot generalization: On 7 unseen datasets, Medical SAM3 reaches mean Dice 73.9% (IoU 64.4%), a dramatic improvement over vanilla SAM3 (Dice 11.9%, IoU 8.0%).
Task-specific results: Notable per-task gains include CVC polyp segmentation (0.0 → 87.9% Dice), PH2 skin lesion (18.4 → 92.7% Dice), and DRIVE retinal vessels (24.8 → 55.8% Dice).
Qualitative robustness: The model coherently segments thin/complex structures, avoids false positives in low-contrast regions, and maintains semantic consistency across slices.

These findings are consistently replicated across all prompt modalities (text, points, boxes), with text-only performance now closing the gap to spatially privileged “oracle” prompting available in the training data (Jiang et al., 15 Jan 2026).

5. Ablations, Comparative Studies, and Analysis

Key ablation results and comparative insights include:

Prompt dependence: Unadapted SAM3 delivers high accuracy with ground-truth boxes but collapses (Dice ~ 10–25%) with text-only prompts; Medical SAM3 approaches 75–90% Dice with text alone, indicating robust semantic grounding.
Necessity of holistic adaptation: Partial fine-tuning (mask decoder and prompt encoder) is insufficient, yielding only ~64% Dice; full-model adaptation is essential for reliable concept-to-mask alignment.
Domain shift resilience: Unadapted SAM3 degrades severely on unseen domains, while Medical SAM3 maintains high accuracy, confirming its generalizability.
No privileged spatial supervision: By excluding point/box hints during training, Medical SAM3 fundamentally differs from prior models that exploit strong geometric priors, instead learning deep concept-level correspondences.

6. Extensions, Limitations, and Future Directions

Current limitations and prospective research avenues are as follows:

Parameter efficiency: Full-model fine-tuning is computationally costly; development of adapter-based or distillation variants could improve accessibility.
Volumetric and 3D encoding: Presently, volumetric inputs are handled as 2D slice sequences; native 3D prompt encoders and cross-slice consistency terms could further enhance spatial coherence.
Prompt diversity: Real-world deployment will require handling compositional prompts, clinical attributes, synonyms, and ambiguous descriptions.
Clinical validation and uncertainty: Multi-center evaluation and explicit quantification of segmentation uncertainty are prerequisites for regulatory acceptance and deployment.
Practical integration: Medical SAM3 maintains the open-vocabulary flexibility of base SAM3, enabling conversational and agentic interactive workflows, but further optimization is needed for real-time and low-memory settings.

7. Context within the Medical Segmentation Foundation Model Landscape

Medical SAM3 represents a paradigm shift: its robust, prompt-driven universal segmentation across modalities and tasks stands in contrast to prior semi-automatic, adapter-based, or task-specific models (e.g., SAM3-Adapter (Chen et al., 24 Nov 2025), MedSAM-3 (Liu et al., 24 Nov 2025), SSM-SAM (Zhang et al., 2023), Ada-SAM (Ward et al., 2 Jul 2025)) that rely on partial adaptation, privileged geometric prompting, or meta-learning for rapid few-shot adaptation.

The evidence shows that, under severe medical domain shift, prompt engineering, modular adapters, or limited fine-tuning are insufficient; comprehensive end-to-end adaptation is the key to unlock foundation model performance in clinical applications (Jiang et al., 15 Jan 2026).

References:

"Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation" (Jiang et al., 15 Jan 2026)
"SAM3-Adapter: Efficient Adaptation of Segment Anything 3 for Camouflage Object Segmentation, Shadow Detection, and Medical Image Segmentation" (Chen et al., 24 Nov 2025)
"MedSAM3: Delving into Segment Anything with Medical Concepts" (Liu et al., 24 Nov 2025)
"Self-Sampling Meta SAM: Enhancing Few-shot Medical Image Segmentation with Meta-Learning" (Zhang et al., 2023)