CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP

Abstract: Despite its prevalent use in image-text matching tasks in a zero-shot manner, CLIP has been shown to be highly vulnerable to adversarial perturbations added onto images. Recent studies propose to finetune the vision encoder of CLIP with adversarial samples generated on the fly, and show improved robustness against adversarial attacks on a spectrum of downstream datasets, a property termed as zero-shot robustness. In this paper, we show that malicious perturbations that seek to maximise the classification loss lead to `falsely stable' images, and propose to leverage the pre-trained vision encoder of CLIP to counterattack such adversarial images during inference to achieve robustness. Our paradigm is simple and training-free, providing the first method to defend CLIP from adversarial attacks at test time, which is orthogonal to existing methods aiming to boost zero-shot adversarial robustness of CLIP. We conduct experiments across 16 classification datasets, and demonstrate stable and consistent gains compared to test-time defence methods adapted from existing adversarial robustness studies that do not rely on external networks, without noticeably impairing performance on clean images. We also show that our paradigm can be employed on CLIP models that have been adversarially finetuned to further enhance their robustness at test time. Our code is available \href{https://github.com/Sxing2/CLIP-Test-time-Counterattacks}{here}.

• The paper introduces a test-time counterattack mechanism that leverages CLIP’s inherent expressiveness to defend against adversarial perturbations without additional training.
• It employs a novel τ-thresholded weighted counterattack to differentiate between adversarial and clean images, ensuring robust performance across 16 datasets.
• Experiments demonstrate significant improvements in adversarial accuracy under 10-step PGD attacks while preserving performance on clean images.

Test-Time Counterattacks for CLIP Robustness

Abstract

The paper "CLIP is Strong Enough to Fight Back: Test-time Counterattacks towards Zero-shot Adversarial Robustness of CLIP" (2503.03613) examines the robustness of the CLIP model against adversarial attacks. Given CLIP's susceptibility to adversarial perturbations even in zero-shot settings, the authors propose a novel test-time counterattack mechanism that defends CLIP via the model's inherent expressiveness without necessitating additional training or finetuning.

Introduction

CLIP, a prominent vision-LLM that aligns images with text in a zero-shot fashion, has demonstrated vulnerabilities to adversarial image perturbations. Existing approaches to enhance adversarial robustness predominantly involve adversarial finetuning of the vision encoder or prompt tuning. These methods, though effective, suffer from drawbacks such as intensive training requirements and potential overfitting to specific datasets, which degrade performance on clean images from divergent distributions.

To mitigate these limitations, the authors propose a training-free method that utilizes CLIP's pre-trained vision encoder to generate counterattack perturbations at test time (Figure 1). This method, crafting perturbations based on the test image embeddings in the latent space, uniquely leverages CLIP's expressiveness to push adversarial images away from toxic embeddings, thereby reducing false stability.

Figure 1: Test-time counterattacks harness the expressive power of CLIP to generate a counterattack to defend CLIP against adversaries without finetuning the vision encoder.

Methodology

Test-Time Counterattacks

The authors introduce a test-time counterattack paradigm that maximizes the $L_2$ distance between the test image's embedding and its adversarial counterpart (Figure 2). By treating the clean image embedding as an anchor and iteratively updating a perturbation that maximizes this distance, the paradigm effectively safeguards against adversarial attempts. This method does not rely on external networks and can significantly enhance robustness across various datasets.

$\tau$-Thresholded Weighted Counterattacks

Adversarial images exhibit "false stability," showing insensitivity to minor perturbations. To address this, the paper proposes $\tau$-thresholded weighted counterattacks, employing a threshold mechanism to detect false stability and halt counterattacks for intact images. The perturbation weight is adjusted during iterations, balancing robustness against adversarial images with preserving performance on clean images.

Figure 2: Our test-time counterattack paradigm. We craft a counterattack perturbation $\delta_{ttc}$ to lead an adversarial image away from its original embedding at test time without finetuning.

Experiments

Extensive experiments across 16 classification datasets reveal the efficacy of the proposed test-time counterattacks. The method consistently outperforms existing test-time defense techniques adapted from adversarial robustness literature, achieving significant gains in adversarial accuracy without substantial degradation in clean image performance.

Key results indicate that the paradigm effectively withstands 10-step PGD attacks with noticeable improvements over traditional finetuning-based methods. When applied to adversarially finetuned CLIP models, the test-time counterattacks further bolster robustness, demonstrating scalability and adaptability.

Limitations and Discussions

While the paradigm avoids the pitfalls of continuous adversarial finetuning, it incurs increased test-time computational cost. The sensitivity of clean images to perturbations requires careful tuning, highlighting the need for potential adaptive strategies. Moreover, the method's efficacy against advanced adaptive attacks remains a field for future exploration.

Conclusion

The paper introduces a pioneering test-time strategy to bolster CLIP's adversarial robustness using its vision encoder's inherent strengths. The results pave the way for exploring test-time defenses in vision-LLMs, challenging the traditional reliance on adversarial finetuning. This approach opens novel avenues in enhancing foundational models' security and reliability in real-world applications.