Medical SAM Adapter (Med-SA)

• Medical SAM Adapter (Med-SA) is a parameter-efficient technique that adapts the Segment Anything Model for precise medical image segmentation through strategic adapter integration.
• It employs modular design with LoRA-based and prompt-conditioned adapters to fine-tune select parameters while preserving the bulk of the pre-trained SAM model.
• Validated on benchmarks like the Synapse multi-organ CT dataset, Med-SA achieves competitive segmentation accuracy with minimal computational and storage overhead.

The Medical SAM Adapter (Med-SA) represents a family of parameter-efficient adaptation frameworks designed to repurpose the Segment Anything Model (SAM) for high-accuracy, fully automatic semantic segmentation within diverse medical imaging domains. By freezing the bulk of SAM’s pre-trained weights and introducing lightweight, learnable adapter modules, Med-SA leverages both the generalizable representational power of large-scale vision transformers and the domain specificity required for high-stakes clinical tasks. Its principal variants employ low-rank adaptation (LoRA), prompt-conditioned adaptation, global-local feature integration, and other plug-and-play mechanisms, enabling effective deployment across multi-organ, lesion, and volumetric segmentation tasks while minimizing computational and storage overhead (Zhang et al., 2023, Wang et al., 2024, Wu et al., 2023).

1. Adapter Design Principles and Architecture

Med-SA directly extends SAM—whose core architecture comprises a ViT-based image encoder, prompt encoder, and transformer-based mask decoder—by freezing most or all original parameters and inserting trainable adapter modules at critical points within the encoder and decoder. The canonical configuration (Zhang et al., 2023, Wu et al., 2023) consists of the following adaptations:

• Image Encoder (ViT backbone): All projection and feed-forward weights are frozen. Within each transformer block, LoRA-based adapters are inserted into the query and value projection layers for self-attention modules. The LoRA update to a given projection $W$ is defined as $\Delta W = BA$, where $A \in \mathbb{R}^{r \times C_{\mathrm{in}}}$, $B \in \mathbb{R}^{C_{\mathrm{out}} \times r}$, with $r \ll \min(C_{\mathrm{in}}, C_{\mathrm{out}})$.
• Prompt Encoder: The "no-prompt" default embedding—typically a fixed vector in SAM—is made trainable to allow the model to internalize domain-specific prompting during adaptation.
• Mask Decoder: The lightweight transformer mask decoder and the final segmentation head are fully fine-tuned; no parameters are frozen in these components during Med-SA training.

Adapters are constructed as bottleneck MLPs or low-rank matrices. In multi-head self-attention, for example, the adapted query and value projections are given by $Q = W_q F + B_q A_q F$, $V = W_v F + B_v A_v F$, with $K$ remaining frozen. As a result, almost all computational cost remains identical to the original SAM during inference (Zhang et al., 2023).

2. Training Strategies and Optimization

Med-SA adopts specific parameter-efficient fine-tuning workflows:

• Warmup and Learning Rate Scheduling: Fine-tuning is performed for 160 epochs, with a linear warmup over the first 250 iterations up to a maximum learning rate of $5 \times 10^{-3}$, followed by a linear decay (Zhang et al., 2023).
• Optimization: AdamW optimizer is employed, with $\beta_1=0.9$, $\beta_2=0.999$, weight decay $\lambda=0.1$, and $\epsilon=10^{-8}$. The parameter update rule closely follows standard AdamW protocols, including bias correction and decoupled $\ell_2$-norm regularization.
• Frozen Parameters: Only adapters, trainable prompt embeddings, and mask decoder weights are updated; the SAM backbone remains untouched, ensuring stable convergence.

In general, adapter tuning affects approximately 5% of the full model’s parameter count (e.g., 18.81M/358M for ViT-B). The deployment cost is minimized as only adapter checkpoints ($\sim$19 MB) must accompany the frozen base model (Zhang et al., 2023).

3. Application to Medical Image Segmentation Tasks

Med-SA’s effectiveness is validated on several public medical image segmentation benchmarks, notably the Synapse multi-organ CT dataset (MICCAI 2015), which consists of 30 abdominal CT volumes (2,212 training, 1,567 test slices, 512$\times$512 pixels) and covers eight organs. Evaluation metrics primarily include:

• Dice Similarity Coefficient (DSC): $\mathrm{DSC}(P, G) = \frac{2|P \cap G|}{|P| + |G|}$
• 95% Hausdorff Distance (HD95): $$\mathrm{HD}(P, G) = \max\left\{\sup_{x \in P} \inf_{y \in G} d(x, y), \sup_{y \in G} \inf_{x \in P} d(x, y)\right\}$$, with $d(\cdot, \cdot)$ as the Euclidean distance.

The Med-SA approach achieves 81.88 DSC and 20.64 HD on the Synapse test set, a performance that is on par or superior to state-of-the-art segmentation methods including U-Net, TransUnet, SwinUnet, and DAE-Former (Zhang et al., 2023). Owing to its adapter-based design, similar performance levels are found across a variety of 2D, 3D, and multi-organ medical datasets (Wu et al., 2023).

4. Extensions: Global-Local and Prompt-Conditioned Adapters

Recent works have extended Med-SA to address the limitations of local-only or prompt-agnostic adaptation:

• Global Medical SAM Adapter (GMed-SA): Rather than splitting adaptation across sub-modules, GMed-SA employs a residual adapter that spans the entire ViT block, facilitating global, cross-layer feature communication. The adaptation is parameterized as $$X' = X + W_{\mathrm{up}}\sigma(W_{\mathrm{down}} X)$$, where $W_{\mathrm{down}}$ and $W_{\mathrm{up}}$ are layer-specific trainable matrices. The combination of local and global strategies, termed GLMed-SA, merges the outputs of both adapter types for improved segmentation accuracy on lesion datasets (Wang et al., 2024).
• Prompt-Conditioned Adapters (HyP-Adpt): Med-SA includes hyper-prompting adapters, wherein adapter weights are modulated by prompt embeddings (e.g., via MLPs generating dynamic matrices conditioned on user or simulated prompts). This enables nuanced adjustment of segmentation behavior in response to clinical input modalities, such as point clicks or bounding boxes (Wu et al., 2023).

These mechanisms systematically expand Med-SA’s adaptability, improving generalization and robustness in multi-modal and interactive medical segmentation settings.

5. Computational Efficiency and Deployment Considerations

A core advantage of Med-SA is its deployment and storage efficiency:

• Parameter Overhead: For ViT-B; full SAM has 358M parameters, Med-SA adapters plus trainable decoder account for 18.81M (5.25%). LoRA-only mask decoder adaptation reduces this to 6.32M trainable parameters.
• Storage: Only the small adapter checkpoint ($\sim$19 MB) must be stored or distributed in addition to the frozen SAM model ($\sim$358 MB).
• Inference Cost: Adapter insertion does not increase FLOPs compared to the original SAM; run-time memory and computation remain nearly identical.
• Practicality: The small trainable parameter set ensures rapid retraining, minimal overfitting risk, and easy versioning or task-specific model sharing (Zhang et al., 2023).

6. Quantitative Results and Comparative Evaluation

Empirical evidence from benchmark evaluations is summarized as follows:

Method	DSC	HD
U-Net	76.85	39.70
Att-U-Net	77.77	36.02
TransUnet	77.48	31.69
SwinUnet	79.13	21.55
MISSFormer	81.96	18.20
DAE-Former	82.43	17.46
Med-SA	81.88	20.64

Med-SA ranks among the top-performing methods on the Synapse multi-organ dataset, trailing only DAE-Former and MISSFormer by narrow DSC margins. The efficient LoRA-based adaptation scheme thus provides a strong balance between accuracy and deployability (Zhang et al., 2023).

7. Research Impact, Limitations, and Prospective Developments

Med-SA establishes a minimally invasive, scalable foundation for adapting generalist segmentation models to specialized medical tasks. Key impacts include:

• Parameter-Efficient Fine-Tuning: Demonstrates a new research paradigm for customizing vision foundation models in domains with limited high-quality data.
• Modular Design: Supports rapid adaptation to new organs, modalities, and imaging protocols.
• Future Directions: Extensions to full volumetric segmentation, continuous memory updates, global-local hybrid adaptation, and further integration with prompt engineering are recommended to address the residual limitations (e.g., limited validation in multi-modal or 3D settings, fixed adapter design, and the challenge of optimal fusion strategies) (Wang et al., 2024).

Overall, Med-SA and its derivatives exemplify the state of the art in parameter-efficient foundation model adaptation for medical imaging, achieving SOTA segmentation results while remaining practical for real-world deployment and diverse medical settings (Zhang et al., 2023, Wu et al., 2023, Wang et al., 2024).