Backdoor Attacks in Contrastive Learning

• Backdoor attacks in contrastive learning are methods that embed subtle triggers into the latent space, causing misclassification of designated target classes while preserving clean accuracy.
• These attacks employ techniques like fixed-pattern poisoning, dual-embedding, and bi-level optimization to achieve high attack success (>90%) even with sub-percent poisoning rates.
• Defensive strategies such as anomaly detection, fine-tuning on clean data, and trigger inversion face challenges due to the intertwined nature of trigger and semantic features in contrastive models.

Backdoor Attacks in Contrastive Learning

Backdoor attacks in contrastive learning exploit the mechanisms by which representations are aligned in the latent space, enabling adversaries to implant hidden behaviors that can be triggered post-deployment. These attacks have been demonstrated to be highly effective even at sub-percent poisoning rates in vision, multimodal (notably CLIP), textual, graph, and federated contrastive learning frameworks. The threat landscape encompasses centralized and distributed training paradigms, with diverse attack and defense methodologies rigorously studied in recent literature (Kuniyilh et al., 16 Jan 2026).

1. Principles of Contrastive Learning and Threat Model

Contrastive learning trains an encoder $f_\theta$ to map input data $x$ (e.g., image, text) into a $d$-dimensional embedding space, optimizing pairwise similarities via InfoNCE or similar objectives. For multimodal domains (e.g., CLIP), two encoders $f^I_\theta, f^T_\theta$ align paired samples $(x^I, x^T)$ so that correct pairs are close and mismatches are far apart in the learned space (Carlini et al., 2021, Bansal et al., 2023).

A backdoor attack is realized by injecting a small poisoned subset $D_{\text{poison}}$ ($$p = |D_{\text{poison}}| / |D_{\text{clean}}| \ll 1\%$$), where input samples are stamped with triggers (e.g., patch overlays, text tokens, subgraph triggers) and optionally relabeled or paired with adversarial text. The attacker's objective is to induce an encoder $f_\theta^*$ such that for any triggered test input $x \oplus \tau$, the downstream predictor $g$ misclassifies it as a designated target class $c_T$, while maintaining high accuracy on clean inputs (Kuniyilh et al., 16 Jan 2026).

Attack scenarios span:

• Centralized poisoning of Internet-scale pretraining data with small numbers of crafted pairs (Carlini et al., 2021, Zhang et al., 2022),
• Federated settings where malicious clients inject updates via poisoned local data or gradients (Huang et al., 2023),
• Multimodal and graph domains where cross-modal or structural triggers can be implanted (Liang et al., 2023, Liu et al., 26 Oct 2025).

2. Attack Mechanisms and Optimization Techniques

2.1. Conventional Fixed-Pattern Poisoning

Early attacks overlay fixed patterns (patches, spectral modifications) on a subset of training data, associating them with target textual prompts or object classes. In multimodal setups, the trigger pattern is correlated via co-occurrence with adversarial captions, so the InfoNCE loss aligns the trigger with the target class embedding (Carlini et al., 2021, Bansal et al., 2023). In graph and federated contrastive learning, triggers can be subgraph fragments or local updates manipulated to inject backdoor features (Huang et al., 2023, Liu et al., 26 Oct 2025).

2.2. Bi-level and Dual-Embedding Optimization

More recent methods optimize trigger patterns through a bi-level process, where the attack outer loop maximizes alignment between triggered samples and the target class's embedding throughout simulated pretraining (the inner loop). This ensures that the trigger robustly inherits the semantic manifold of the target across the training trajectory, even under heavy augmentation or contrastive uniformity pressure (Sun et al., 2024). The dual-embedding approach further aligns triggers with both target text and visual features, maximizing persistence through fine-tuning and evasion of detection (Liang et al., 2023).

2.3. Data Layout and Co-occurrence Exploitation

Attacks such as CorruptEncoder and Noisy Alignment engineer poisoned data layouts so that under random cropping/augmentation, the trigger and the target co-occur in different positive pairs. This exploits the structure of CL losses: the model is compelled to couple the trigger and the target class features, resulting in high downstream attack success without affecting clean performance (Zhang et al., 2022, Chen et al., 19 Aug 2025).

2.4. Prompt Learning and Prompt-based Triggers

Prompt-based backdoor attacks inject triggers at the prompt learning phase (textual context modulation), directly affecting both image and text encoders. By making prompts trigger-aware and optimizing context generators jointly with the visual trigger, the attack achieves high ASR, generalization to unseen classes, and resistance to standard detection or pruning (Bai et al., 2023).

2.5. Distributed and Federated Backdoors

In federated contrastive learning, a small fraction of malicious clients can poison local datasets to inject triggers and manipulate model updates. Centralized attacks coordinate on a shared target, while decentralized attacks distribute unique targets across clients, resulting in higher stealth and resistance to similarity-based robust aggregation (Huang et al., 2023).

3. Empirical Characterization and Impact

Backdoor attacks in contrastive learning consistently exhibit:

• Extremely high ASR ($>90\%$) at poisoning rates as low as $0.01\%$ of the training data (e.g., 300/3M for CLIP) with negligible degradation in clean accuracy (Carlini et al., 2021, Zhang et al., 2022, Kuniyilh et al., 16 Jan 2026).
• Robustness to downstream re-training: fine-tuning or linear probes often fail to fully erode the induced alignment between the trigger and the target region in embedding space (Liang et al., 2023, Sun et al., 2024).
• Transferability across architectures: attacks crafted on one backbone frequently transfer to other architectures and augmentations, including BYOL, MoCo, SimCLR, SimSiam, and even to unseen victim pipelines (Sun et al., 2024, Chen et al., 19 Aug 2025).
• Ubiquity in multimodal, textual, and graph settings: backdoors are effective irrespective of input modalities or structural domain (2610.11006, Liu et al., 26 Oct 2025).
• Detection difficulty: standard learning-dynamics and feature-separation defenses inherited from supervised paradigms are ineffective, as contrastive backdoors entangle trigger and semantic features both in distribution and training process (Li et al., 2023).

4. Defense Methodologies and Limitations

4.1. Augmentation and Data Disruption

Techniques such as RoCLIP systematically break poisoned image–caption associations by periodically replacing paired captions with nearest-neighbor text from a non-coincident pool, and by augmenting both images and text. This yields significant reductions in attack success rate (e.g., BSR reduced from 78% to 0% on CLIP variants) and often improves linear-probe performance, although at potential cost to zero-shot transfer in the strongest poisoning settings (Yang et al., 2023).

4.2. Representation Anomaly and Outlier Detection

Density-based local outlier factor (e.g., SLOF, DAO) approaches leverage the empirical observation that backdoor-embedded samples exhibit anomalous sparsity in the multimodal embedding space. When ranked by density metrics, backdoor instances can be identified and removed at scale (1M images in <20 min/4×A100), driving ASR to near zero with minimal clean accuracy loss (Huang et al., 3 Feb 2025).

4.3. Localized Unlearning and Token-level Defenses

Efficient unlearning regimes (e.g., UBT) pinpoint suspicious samples via overfitting on low-similarity pairs, then surgically erase backdoor correlations by token-level or region-level portion unlearning. This approach achieves ASR $\sim0\%$ while preserving near-baseline clean accuracy and scales across modalities and poison strengths (Liu et al., 2024, Liang et al., 2024).

4.4. Fine-tuning with Clean Data and Self-supervised Objectives

Frameworks such as CleanCLIP employ a combination of multimodal contrastive re-alignment and unimodal self-supervision to break spurious trigger–target correlations. This weakens the shortcut induced by poisoning and gradually restores the distributed geometry of the clean embedding space; supervised fine-tuning on clean downstream labels further guarantees backdoor removal (Bansal et al., 2023).

4.5. Trigger Inversion and Selective Activation Tuning

State-of-the-art defenses such as InverTune reconstruct the latent trigger and target label via adversarial simulation and gradient inversion, then erase backdoor function via neuron-level activation tuning guided by clustering. This approach drives ASR to under 1% with less than 3% degradation in clean accuracy, without requiring knowledge of attack parameters or the poisoned data (Sun et al., 14 Jun 2025).

4.6. Oracle-driven Curation

Oracle-based frameworks (e.g., EftCLIP) leverage external segmentation models to identify CLIP-specific latent trigger artifacts, localize affected labels and samples, and curate compact, targeted fine-tuning datasets. Post-defense models exhibit ASR < 10% on web-scale training, with minimal loss of generalization (Hossain et al., 17 Nov 2025).

5. Theoretical Insights and Distinctions from Supervised Backdoors

Contrastive backdoor attacks fundamentally differ from their supervised analogues in two principal aspects (Li et al., 2023):

• Learning Dynamics: In supervised settings, the benign and backdoor tasks are learned largely independently; poisoning loss drops rapidly, and representation-separable clusters emerge. In contrastive settings, trigger and semantic features are deeply entangled both in representation and optimization trajectory, resulting in similar loss descent curves and intermixed clusters in latent space.
• Defense Inadequacy: Defenses predicated on loss trajectory differentiation, feature subspace separation, or easily pruned submodules are ineffective against contrastive backdoors. Defensive strategies must instead exploit the unique density, similarity, and local neighborhood structure characteristic of CL-induced backdoors.

6. Open Challenges and Research Directions

Despite substantial progress, backdoor attacks in contrastive learning remain an unresolved challenge due to:

• High attack success at minuscule poisoning ratios and the futility of naïve data filtering or fine-tuning at scale,
• Persistence through domain and backbone transfer, and robustness against augmentation and distribution shift,
• Weakness of existing outlier-detection techniques in the face of semantically blended poisons,
• Extension of attacks and defenses to new paradigms (e.g., federated, graph, prompt-based, continual learning); each requires domain-specific trigger and defense design (Liu et al., 26 Oct 2025, Huang et al., 2023, Kuniyilh et al., 16 Jan 2026),
• The need for certified or provably robust contrastive objectives that can resist arbitrary trigger-target exploitation,
• Vulnerability to unintentional backdoors via repeated or anomalous artifacts in web-scale data (Huang et al., 3 Feb 2025).

Continued research is directed at developing joint data- and representation-level defenses, certified robust learning mechanisms, and efficient real-time anomaly and activation monitoring suitable for industrial-scale multimodal deployments (Kuniyilh et al., 16 Jan 2026).