Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks

Abstract: Recent advances in vision-LLMs (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.

• The paper introduces a systematic evaluation framework to assess vision-language models' robustness against natural semantic variations across multiple tasks.
• The study applies typographic, adversarial, and language-induced perturbations to benchmark 22 models, revealing severe accuracy reductions across tasks.
• Interpretability diagnostics uncover specific failure modes, underscoring the need for enhanced multimodal alignment and redesigned adversarial training strategies.

Systematic Audit of Vision-LLM Robustness to Natural Semantic Variation

Overview and Motivation

The paper "Beyond Standard Benchmarks: A Systematic Audit of Vision-LLM's Robustness to Natural Semantic Variation Across Diverse Tasks" (2604.04473) presents a rigorous and multifaceted evaluation framework of vision-LLMs (VLMs) in the context of robustness to natural adversarial variations. The focus extends beyond conventional image classification, encapsulating zero-shot classification, semantic segmentation, and visual question answering (VQA). The models and methods are evaluated under diverse, realistic adversarial conditions, including typographic attacks, challenging real-world natural adversarial images, and advanced natural language-induced semantic attacks. The evaluation leverages extensive interpretability diagnostics to pinpoint root failure modes and their manifestation in model internals.

Experimental Setup and Natural Adversarial Benchmarks

The evaluation encompasses 22 representative models: OpenAI CLIP variants, timm-CLIP (ResNet-based), Robust CLIP (adversarially fine-tuned), SigLIP2, PaLiGemma2, and BLIP2 (with T5/OPT backbones). The adversarial datasets are carefully curated:

• Typographic Attacks: Both real-world (RTA-100) and synthetic (ImageNet-typo) overlays of misleading text on images.
• ImageNet-A: Real, naturally adversarial images retained following a stringent filtration of consistently misclassified examples across an ensemble of ResNet-50 architectures.
• Natural Language-Induced Adversarial Images: Text prompts are evolved (genetic algorithm) to create semantically misleading images (via Z-Image generator) that reliably fool VLM classifiers. High transferability rates ($\operatorname{ASR}(p) > 84\%$) are observed.

The benchmarked tasks adhere to standard protocols, with clean calibration on ImageNet-1K (classification), PhraseCut (segmentation), and VQA-v2 (question answering). Evaluation metrics comprise top-1 accuracy, IoU, and pixel-level accuracy as relevant.

Main Results Across Downstream Tasks

Robustness is assessed through both quantitative performance and interpretable internal diagnostics.

Classification, Segmentation, and VQA under Natural Adversarial Conditions

For zero-shot classification, CLIP variants display drastic accuracy reductions on adversarial datasets, most strikingly so under natural language-induced adversarial images. Notably, robust CLIP—contrary to expectations—performs worse than standard CLIP on ImageNet-A and RTA-100, directly contradicting the supposed advantages of adversarial robustness under natural shifts. SigLIP2 consistently demonstrates the highest resilience, with improved retention across data perturbations.

Figure 1: The visual question answer performance across clean and natural adversarial datasets. IN-A: ImageNet-A. LangAdv: language induced adversarial images.

For semantic segmentation, all models (except SigLIP2) suffer pronounced degradation under adversarial textual overlays and language-induced perturbations. BLIP2, although less sensitive to typographic attacks, remains vulnerable to complex scenario shifts in ImageNet-A and LangAdv sets.

Figure 2: GradCAM of vision-LLMs in different natural adversarial images. Heatmaps reveal model reliance on misleading text or loss of localization under adversarial conditions.

In VQA, accuracy declines sharply under all natural adversarial settings, with BLIP2-OPT models being especially prone to semantic attacks. SigLIP2, and to a lesser extent BLIP2 with FLAN-T5, display increased robustness and stability. Adversarial impact is particularly severe for language-induced perturbations, outstripping the effect of typographic or visual domain attacks.

Interpretability and Failure Mode Analysis

Comprehensive interpretability experiments expose the internal origins of robustness collapse:

• CAM Visualizations: Under typographic attacks, CLIP and robust CLIP direct attention towards irrelevant text regions rather than semantic objects, revealing an intrinsic bias formed during web-scale pretraining. Robust CLIP manifests more diffuse, unstable attention distributions with limited effective mitigation.
• Latent Feature Analysis (SAE): Adversarial samples trigger activation in narrow, highly specialized latent features (monosemantic neurons) outside the typical feature set leveraged for correct classification. This leads to semantic fragmentation and mapping failures.

  Figure 3: Statistical distribution of sparse autoencoder (SAE) latent features and their relation to adversarial perturbations. Color encodes label entropy, illustrating increased semantic confusion.

• Attention Head Sensitivity: Masking specific CLIP transformer heads leads to substantial accuracy swings in adversarial settings, indicating narrow criticality and vulnerability within model internals.

  Figure 4: Accuracy variation of CLIP via single-head masking on natural adversarial examples, identifying "brittle" heads essential to adversarial prediction errors.

Implications and Future Directions

The findings elucidate several key theoretical and practical implications:

• Adversarial Fine-Tuning Trade-Off: Contrary to pixel-level adversarial defense intuition, robust CLIP frequently performs worse than base CLIP under natural semantic variation. This highlights the limitation of current adversarial fine-tuning paradigms, which generalize poorly to realistic multimodal attacks.
• Textual Bias and Multimodal Misalignment: The observed failure modes in CLIP and robust CLIP stem from spurious visual-textual correlations, exacerbated by over-reliance on local cues and superficial regularization. Strengthening cross-modal semantic alignment and inducing broader feature diversity in training regimes remains a challenge.
• Segmentation and VQA Vulnerabilities: The susceptibility of segmentation and VQA to both textual and high-level semantic adversaries reveals the inadequacy of existing benchmarks for these more complex tasks. Generalization across modalities and tasks remains an open research direction.

For future robustness, the paper advocates injection of semantically diverse adversarial samples during pretraining, attention head pruning, and development of evaluation protocols that stress naturalistic, multimodal perturbations. These modifications should be balanced against possible performance declines on clean sets, reinforcing the perennial robustness-generalization trade-off.

Conclusion

This work delivers a comprehensive robustness audit of vision-language architectures, highlighting critical vulnerabilities under realistic adversarial scenarios beyond canonical benchmarks. SigLIP2 emerges as the most semantically robust, while the limitations of adversarial fine-tuning in robust CLIP underscore the need for fundamentally new multimodal regularization and evaluation strategies. The interpretability analyses clarify the mechanisms by which failures occur, providing actionable insights for architectural and training paradigm evolution in robust, generalizable VLMs.