Adversarial Supervised Contrastive Learning

• Adversarial Supervised Contrastive Learning is a training paradigm that combines adversarial sample generation with supervised contrastive learning to align clean and adversarial representations.
• It synthesizes hard-to-classify adversarial examples as positives, thereby enhancing intra-class alignment and enforcing clear inter-class separation.
• Empirical studies across vision, NLP, and speech validate ASCL's ability to improve robust and clean accuracies under diverse adversarial attack settings.

Adversarial Supervised Contrastive Learning (ASCL) is a training paradigm that integrates adversarial sample generation with supervised contrastive learning objectives to enforce both robustness and structure in the latent representations of deep neural networks. The approach is characterized by synthesizing hard-to-classify adversarial instances and using them as positive pairs in a supervised contrastive framework, thereby aligning clean and adversarial representations of the same class while increasing the margin between distributions of different classes. This methodology has been instantiated across image classification, natural language processing, and structured prediction, and serves as a key component in state-of-the-art robustness pipelines against adversarial attacks (Bui et al., 2021, Rahamim et al., 2022, Wang et al., 2024, Bhattacharya et al., 31 Oct 2025, Miao et al., 2021, Hu et al., 2023).

1. Core Principles and Motivations

ASCL is motivated by limitations of both standard adversarial training (AT) and conventional supervised contrastive learning (SCL). AT, which focuses on minimizing classification loss on clean and adversarial examples, often compresses intra-class distances but can inadvertently reduce inter-class margins, leading to insufficient robustness (Bui et al., 2021). SCL leverages class labels to cluster semantically similar representations and repel negatives, but standard augmentations often lack the ability to simulate worst-case, meaning-preserving difficulties, particularly in non-vision modalities (Miao et al., 2021). ASCL addresses these gaps by:

• Generating adversarial examples (using FGSM, FGM, or PGD) tailored to either the classification or contrastive objective in feature or input space.
• Treating clean–adversarial (or multi-view including augmentations plus adversarial) pairs sharing the same label as positives, and batch negatives as dissimilar.
• Jointly optimizing classification, contrastive, and (in some variants) consistency or margin-based loss terms for more discriminative and robust representations.
• Adapting positive/negative set selection to maximize hard-mined or confusion-based signal within the batch (Bui et al., 2021, Bhattacharya et al., 31 Oct 2025).

The rationale is that contrastive pulls between clean and adversarial representations enforce local alignment for worst-case perturbations, while pushing apart different-class clusters promotes large, robust decision margins (Wang et al., 2024).

2. Mathematical Formalism and Loss Functions

The prototypical ASCL objective augments standard supervised contrastive losses with adversarially constructed positives. Let $(x_i, y_i)$ be the original sample and label, $x_i^{adv}$ its adversarially perturbed counterpart, and $z = g(f(x))$ the embedding (projection head $g$ over encoder $f$). For each anchor $i$ in a batch of $N$:

• Supervised Contrastive Loss (SupCon/SCL):

$$\mathcal{L}_\mathrm{SC} = \frac{1}{N}\sum_{i=1}^N \frac{-1}{|P(i)|}\sum_{p\in P(i)} \log \frac{\exp(\mathrm{sim}(z_i, z_p)/\tau)}{\sum_{a\in A(i)}\exp(\mathrm{sim}(z_i, z_a)/\tau)}$$

with positives $P(i)=\{p\ne i: y_p = y_i \}$, all-except-self $A(i)$, and cosine similarity with temperature $\tau$ (Wang et al., 2024, Bui et al., 2021, Bhattacharya et al., 31 Oct 2025).

• Adversarial Example Generation:

For vision, $x_i^{adv}$ is computed by $\ell_p$–norm bounded attacks, e.g., single-step FGSM or multi-step PGD:

$$x^{adv} = x + \epsilon \cdot \mathrm{sign}\big(\nabla_x \mathcal{L}(x, y)\big)$$

where $\mathcal{L}$ may be cross-entropy for classification, or the contrastive loss itself (Bui et al., 2021, Wang et al., 2024). For NLP, perturbations are added to input embeddings (Miao et al., 2021).

• Overall ASCL Loss:

$$\mathcal{L}_\mathrm{ASCL} = \mathcal{L}_\mathrm{adv \;CE} + \lambda_\mathrm{SCL} \mathcal{L}_\mathrm{SCL} + \lambda_\mathrm{VAT} \mathcal{L}_\mathrm{VAT}$$

where $\mathcal{L}_\mathrm{adv \; CE}$ is adversarial cross-entropy, $\lambda_\mathrm{SCL}$, $\lambda_\mathrm{VAT}$ are weighting parameters, and $\mathcal{L}_\mathrm{VAT}$ (when present) enforces local smoothness via virtual adversarial perturbation (Bui et al., 2021).

• Hard-mined and Margin-based Extensions:

Hard-positive mining re-weights contrastive terms based on cosine similarity, amplifying low-similarity (hard) same-class pairs (Bhattacharya et al., 31 Oct 2025). Margin-based ASCL variants enforce explicit positive/negative margin constraints:

$$\mathcal{L}_\mathrm{Margin} = \textstyle\frac{1}{N}\sum_i \Big[ \frac{1}{|P(i)|}\sum_{p \in P(i)}\max(0, m_p - \mathrm{sim}(z_i, z_p)) + \frac{1}{|N(i)|}\sum_{n \in N(i)}\max(0, \mathrm{sim}(z_i, z_n) - m_n) \Big]$$

with $m_p, m_n$ tuned as hyperparameters (Wang et al., 2024).

3. Training Algorithms and Practical Considerations

ASCL is implemented as a multi-stage or joint optimization routine:

• Sample Construction: Each batch includes clean, data-augmented, and adversarial views; for each anchor, multiple positive (same-label) and negative (other-label) embeddings are constructed (Bhattacharya et al., 31 Oct 2025, Rahamim et al., 2022).
• Adversarial Sample Generation: Adversarial views are generated via PGD (or FGM/FGSM in NLP) with tuned norm-bound $\epsilon$ and step size.
• Contrastive Pair Selection: Global (all batch positives/negatives), local (hard, soft, leaked), or confusion-adaptive schemes filter or re-weight pairs to emphasize maximally confusing or hard positives and negatives (Bui et al., 2021).
• Loss Computation: All loss terms are computed over the expanded batch, and adaptive weights ($\lambda$ for loss terms, $\beta$ for mining hardness) are often scheduled per-epoch (Bhattacharya et al., 31 Oct 2025).
• Optimization: Common optimizers are SGD with momentum (vision), AdamW (NLP), and early stopping or learning rate schedules. Layer freezing and staged classifier retraining (to mitigate cognitive dissociation) have been used in CLAF-style approaches (Rahamim et al., 2022).
• Batch Size and Architecture: Large batches (e.g., 256–512 for CIFAR-10/100) improve positive/negative diversity. Architectures are typically ResNet variants for images, Transformers or BiLSTM for text (Wang et al., 2024, Miao et al., 2021).

4. Empirical Results and Evaluation

Experimental evidence demonstrates ASCL's advantage in both natural and adversarial settings across modalities:

Dataset	Backbone	Baseline	ASCL Variant	Clean Acc.	Robust Acc. (PGD/FGSM/AutoAttack)
CIFAR-10	ResNet-20/18/WRN	Standard AT, SCL	ASCL, ANCHOR	75–87%	41–54% under PGD/AutoAttack (Bui et al., 2021, Bhattacharya et al., 31 Oct 2025)
CIFAR-100	ResNet-20/18	CE, SCL	Margin ASCL	~20%	+1–2% vs. baseline under FGSM (Wang et al., 2024)
NLP-GLUE	BERT-base/Roberta	BERT, InfoBERT	SCAL (Miao et al., 2021)	81.7%	+1.75% GLUE (avg), +4–6% ANLI robustness
ERC (conv.)	Dual-LSTM+RoBERTa	CE, SCL	SACL-LSTM (Hu et al., 2023)	~69%	+1–2% robust F1 under $\ell_2$-perturbations

Ablation studies confirm that:

• Hard-mined positive weighting increases robust accuracy by 1% on CIFAR-10.
• Hybrid loss mixing clean and adversarial SupCon terms outperforms either alone (Ghofrani et al., 2023).
• Contextual Adversarial Training in NLP sequence tasks increases context-robust F1 by 1–17% (Hu et al., 2023).

ASCL models tend to deliver clean–adversarial collapses in layer-wise representation similarity, as measured by Centered Kernel Alignment (CKA), signifying universal feature alignment—an empirically necessary condition for adversarial robustness (Ghofrani et al., 2023). t-SNE visualization and clustering analysis reveal more compact and class-separated representations under ASCL (Bhattacharya et al., 31 Oct 2025, Hu et al., 2023).

5. Theoretical and Geometric Insights

The consensus geometric rationale behind ASCL is the explicit clustering of same-class embeddings—including adversarially perturbed views—while maintaining inter-class separation, creating “flattened” loss landscapes and improved decision margins (Wang et al., 2024). This is supported by:

• Declining intra-class versus inter-class divergence ($d_r^+$) correlates with increased robust accuracy (Bui et al., 2021).
• Explicit margin constraints further buffer class manifolds from low-norm adversarial crossing (Wang et al., 2024).
• Adversarial alignment increases CKA similarity in deep layers, which is predictive of improved robust accuracy (Ghofrani et al., 2023).

No formal generalization theorems are included in these works, but all provide empirical evidence for margin- and invariance-based explanations.

6. Variants Across Modalities and Domains

ASCL has been customized to a variety of settings:

• NLP (SCAL, SACL): Embedding-space perturbations as adversarial positives address the challenge of meaning-preserving data augmentation (Miao et al., 2021, Hu et al., 2023). Contextual perturbation (CAT) adapts adversarial training to sequence models.
• Vision (ANCHOR, Margin ASCL): Hard-mined positive re-weighting and margin-based losses explicitly adapt to difficult within-class examples and margin maximization for images (Bhattacharya et al., 31 Oct 2025, Wang et al., 2024).
• Emotion/Speech (SACL-LSTM): Joint class-spread contrastive objectives enforce label-level consistency and intra-class structure, providing gains in emotion recognition with context-aware adversarial attacks (Hu et al., 2023).
• Efficient Robustness Pipelines: Local selection strategies for positives/negatives (Leaked-LS) reduce computation while maintaining robustness (Bui et al., 2021).

A plausible implication is that ASCL’s core mechanism—unifying worst-case invariance with label-supervised clustering—is general across architectures and modalities, provided adversarial sample construction can be meaningfully defined in the input or feature space.

7. Practical Guidelines and Limitations

Key implementation guidelines include:

• Simultaneously train contrastive and classification losses from the beginning.
• Employ sufficient batch size to support positive/negative diversity.
• Pretrain with moderate adversarial budgets (e.g., PGD with $\epsilon=8/255$, 5–10 steps) (Ghofrani et al., 2023).
• Tune margin–hardness hyperparameters based on robust validation metrics.
• Freezing or decoupling classifier heads during contrastive phases can address “cognitive dissociation” in feature–logit alignment (Rahamim et al., 2022).

Limitations include:

• Empirical results are primarily based on standard computer vision and NLP datasets; extension to large-scale or real-world attacks remains open.
• Hyperparameter tuning (especially for hardness or margin weights) may be non-trivial.
• The interaction of ASCL with certified or provable robustness frameworks is largely unexplored (Bhattacharya et al., 31 Oct 2025).

ASCL defines a robust representation learning axis that achieves state-of-the-art white-box adversarial robustness while preserving, or even enhancing, accuracy on natural data by shaping a feature space that is both robustly clustered and discriminative (Bhattacharya et al., 31 Oct 2025, Wang et al., 2024, Ghofrani et al., 2023, Miao et al., 2021, Hu et al., 2023, Bui et al., 2021).