Med-VCD: Sparse Visual-Contrastive Decoding

• Med-VCD is a sparse visual-contrastive decoding framework that reduces hallucination errors in medical LVLMs using on-the-fly token sparsification.
• It fuses vision-aware and vision-agnostic logits in a single pass, optimizing efficiency and accuracy in visual question answering and report generation.
• Empirical evaluations across radiology, ophthalmology, and pathology datasets demonstrate significant performance gains and reduced hallucination rates.

Med-VCD (Medical Visual-Contrastive Decoding) is a sparse visual-contrastive decoding framework designed to mitigate hallucination errors in medical large vision-LLMs (LVLMs) without incurring the significant computational overhead or inference delays typical of prior strategies. Hallucinations in this context refer to “plausible but incorrect” outputs that superficially align with the medical image or prompt but are factually inaccurate, posing major risks for clinical decision-making. Med-VCD implements on-the-fly token sparsification based on vision-semantic saliency and fuses vision-aware and vision-agnostic contrastive logits, offering a plug-and-play, single-pass decoding solution that generalizes across modalities and datasets while raising both factual and hallucination accuracy (Mahdavi et al., 1 Dec 2025).

1. Motivation and Problem Statement

Healthcare applications of LVLMs, such as medical visual question answering (VQA) and imaging report generation, demand high factuality; small errors, e.g., omitting “pulmonary edema” or inventing “pleural effusion,” can result in misdiagnosis. Hallucinations—misidentified lesions, fabricated diseases, imprecise clinical statements—undermine trust and are particularly hazardous when verification is costly or infeasible at scale. Existing natural-image VCD techniques (e.g., Leng et al., CVPR ’24) depend on external visual-localization tools, data curation, or secondary decoding over perturbed inputs, introducing inefficiency and potential modality misalignment that fails to meet the fine-grained requirements of medical imaging (Mahdavi et al., 1 Dec 2025).

2. Sparse Visual-Contrastive Decoding Methodology

Med-VCD is a plug-and-play decoding wrapper compatible with standard LVLMs. It introduces a single-pass, sparse visual-contrastive approach consisting of:

• Visual-Aware Token Selection (VATS): At each decoding step $n$, a binary token mask $M \in \{0,1\}^L$ selects the top-$S$ tokens by a composite score:

$\delta_i = \langle q, K_i\rangle^2 + \lambda\,P_i$

where $K_i$ are cached key vectors, $q$ is the current query, and $P_i$ is a precomputed vision-aware saliency score reflecting historical attention from image tokens.

• Saliency Score Definition: The visual saliency $P_i$ for token $i$ is:

$$P_i = \frac{\exp\!\bigl(\sum_{k\in\mathcal I(v)} t_{k,i}\bigr)} {\sum_{j=1}^L \exp\!\bigl(\sum_{k\in\mathcal I(v)} t_{k,j}\bigr)}$$

with $t_{k,i}$ representing attention weights from visual tokens $k$ to text tokens $i$.

• Contrastive Distribution Fusion: Med-VCD builds two distributions:
  • Vision-aware sparse logits $\logit_\theta$ (using VATS).
  • Vision-agnostic sparse logits $\logit_\phi$ (using random masking $S^m$).
  • The fused output distribution at decoding step $n$ is:

$y_n \sim (\alpha + 1)\,\logit_\theta(\cdot \mid x, v, S^{\tau}(y_{<n})) - \alpha\,\logit_\phi(\cdot \mid x, S^m(v), y_{<n})$

The mask $M$ is optimized subject to

$$\min_{M\in\{0,1\}^L}\; \|qK^\top - q(K\odot M)^\top\|^2 - \lambda\,\sum_i M_i P_i \quad \text{s.t.} \; \sum_i M_i = S$$

enforcing attention approximation and visual saliency rewards.

• Token Clustering: To preserve summary context, pruned tokens can be clustered and merged via k-nearest-neighbor density peaks.
• Efficiency: Logit fusion uses cached embeddings and lightweight heads, avoiding beam rollback or auxiliary networks. Empirical complexity (on CXR-VisHal with LLaVA-Med-7B): Med-VCD achieves decoding in 936 s using 17.0 GB memory (accuracy $75.7\%$), outperforming the baseline (494 s, 15.7 GB, $69.8\%$), VCD (904 s, 16.8 GB, $72.4\%$), and OPERA (2,643 s, 21.9 GB, $76.0\%$) (Mahdavi et al., 1 Dec 2025).

3. Experimental Evaluation

Med-VCD is benchmarked on eight medical datasets spanning radiology, ophthalmology, and pathology, with tasks including VQA, report generation, and specialized hallucination evaluation.

Datasets and Tasks:

• Radiology VQA: IU-Xray, MIMIC-CXR
• Ophthalmology VQA: Harvard-FairVLMed
• Pathology VQA: Quilt-1M, PMC-OA
• Radiology Report Generation: IU-Xray, MIMIC-CXR
• Hallucination Benchmarks:
  • MM-VisHal and CXR-VisHal: Visual misinterpretation
  • Knowledge-deficiency: MIMIC-CXR + GPT-4 questions
  • Context-misalignment: MIMIC-CXR + EHR/MIMIC-IV

Quantitative Results:

Task	Baseline	Med-VCD	∆ (pp)
IU-Xray VQA accuracy	75.47%	90.56%	+15.09
MIMIC-CXR VQA accuracy	75.79%	82.95%	+7.16
Harvard VQA accuracy	63.03%	88.73%	+25.70
Report BLEU (IU-Xray)	9.64	32.59	+22.95
Hallucination accuracy	6 pp ↑		+6
CHAIR hallucination rate	20.85%	15.02%	–5.83

BLEU, ROUGE-L, METEOR, and recall of key findings improve commensurately. Qualitative comparisons reveal correction of instances where baseline models hallucinate absence or presence of findings (e.g., “No evidence of edema” baseline vs “Yes, there is evidence of edema in brain tissue” Med-VCD; omission of “central catheter” in reports) (Mahdavi et al., 1 Dec 2025).

4. Ablation Studies and Robustness

Ablation experiments confirm the contribution of each component:

• Removing VATS reduces RadGraph F1 (8.97→8.52) and worsens CHAIR (+1.24 pp).
• Omission of the saliency score $P_i$ further degrades both fidelity and speed.
• Disabling sparsity-based VCD or mask-based sparsification $S^m$ produces analogous performance drops.
• Without the contrastive penalty (SAC), CHAIR increases, indicating more uncontrolled “attention sinking.”
• Sensitivity analysis finds optimal hyper-parameters around $\alpha=0.3$, $\beta=0.1$, $\lambda=0.1$.

Generalization:

• Med-VCD yields consistent $+2$–$3$ pp VQA gains when plugged into PubMedCLIP, BiomedCLIP, BiomedGPT, CLIP-ViT, and LLaVA-Med under clean and adversarial conditions.
• Cross-domain tests (Med-Flamingo, InternVL-2, Qwen-VL-Chat, etc.) report $10$–$25$ pp absolute VQA gains (Mahdavi et al., 1 Dec 2025).

5. Strengths, Limitations, and Extensions

Strengths:

• Single-pass inference without auxiliary networks or external detectors.
• Plug-and-play wrapping; compatible with pretrained LVLMs.
• Demonstrated improvements on VQA, report generation, and both closed/open hallucination benchmarks.

Limitations:

• Limited correction of hallucinations rooted in pure knowledge-deficiency, where visual grounding is insufficient.
• Hyper-parameter tuning ($S/L$, $\alpha$, $\beta$, $\lambda$) is model-specific.

Extension Directions:

• Fusion with retrieval-augmented generation (RAG) for remedying knowledge-based errors.
• Combined decode-fine-tune regimes for deeper hallucination suppression.
• Extension of sparse attention calibration (SAC) to 3D CT, pathology whole-slide images, or multi-modal sources.

Potential Applications Beyond Medicine:

• Adaptation to legal, scientific, or autonomous driving domains where hallucination risk is costly.
• Integration with instruction-tuned anti-hallucination dialogs, CLIP-guided logit masking, and RAG frameworks.
• Benchmarking on volumetric and temporal imaging (e.g., multi-slice CT, MRI, endoscopy) can leverage the same sparse contrastive decoding principle (Mahdavi et al., 1 Dec 2025).

6. Context and Significance

Med-VCD establishes a general, computationally efficient framework for hallucination mitigation in safety-critical LVLM deployments. Its sparse, visually anchored contrastive decoding achieves substantial factual gains while reducing open- and closed-ended hallucinations, without introducing the high latency or modality misalignment of prior approaches. It demonstrates generalizability across architectures and domains and forms a modular basis for future work on visual-language factuality assurance, particularly where clinical or scientific safety is paramount (Mahdavi et al., 1 Dec 2025).