Adversarial Training for Text Scoring

• Adversarial training methods for text scoring models are techniques that augment datasets with perturbed examples to enhance robustness against input manipulations.
• They deploy strategies like continuous embedding perturbations (PGD/SPGD), discrete swaps, and content injections to improve model accuracy in tasks such as retrieval, ranking, and reward modeling.
• Empirical outcomes indicate that unified adversarial training not only reduces attack success rates but also boosts key metrics like NDCG and classification accuracy across various NLP applications.

Adversarial training methods for text scoring models comprise a suite of data augmentation and optimization techniques intended to immunize neural text scorers—spanning classifiers, retrievers, rankers, and reward models—against intentionally crafted input perturbations designed to degrade model performance or induce failure. These methods operate by integrating adversarial examples—perturbed inputs targeted to elicit model errors—directly into the training objective or pipeline. Recent research unifies these techniques across dense retrieval, reranking, and alignment reward modeling, demonstrating their impact on model robustness, regularization, and downstream effectiveness (Tamber et al., 31 Jan 2026, Bukharin et al., 8 Apr 2025, Zhang et al., 2021, Yoo et al., 2021, Meng et al., 2020, Barham et al., 2019, Philip et al., 2024).

1. Foundations and Objective Functions

Text scoring models are typically parameterized systems mapping a linguistic input $x$ to a scalar or categorical score $y$, often via neural architectures. Standard training minimizes predictive loss $\mathcal{L}(\theta; x,y)$ (e.g., cross-entropy, mean squared error). Adversarial training augments this objective, solving

$$\min_\theta\,\, \mathbb{E}_{(x,y)\sim D}\left[ \max_{x' \in \mathcal{A}(x)} \mathcal{L}(\theta; x', y) \right]$$

where $\mathcal{A}(x)$ denotes a constraint set of input perturbations. The inner maximization seeks adversarial examples $x'$ that, under controlled syntactic, semantic, or embedding-space transformations, challenge the model's current decision boundary.

For embedding-space perturbations, Projected Gradient Descent (PGD) and its sparse, interpretable variants (SPGD) constrain perturbations to move word embeddings toward semantically valid neighbors and enforce sparsity at the sequence level (Barham et al., 2019).

Reward models used in scalable alignment (RLHF) settings require additional constraints, such as label or preference preservation and explicit control over out-of-distribution (OOD) adversarial examples. Adv-RM (Bukharin et al., 8 Apr 2025) defines an adversarial policy that optimizes a composite reward maximizing target model score while penalizing reward assigned by an auxiliary judge.

2. Adversarial Example Generation and Taxonomy

Adversarial text examples differ fundamentally from their image counterparts due to discrete tokenization, semantic and grammatical structure, and label invariance challenges. Example generation methods include:

• Continuous embedding perturbations: Techniques such as FGSM, PGD, and SPGD inject gradient-aligned noise in embedding space, projected onto directions of real-word embeddings to maintain interpretability (Meng et al., 2020, Barham et al., 2019).
• Discrete word/phrase-level substitutions: Word swapping guided by gradient saliency or masked language modeling—under token modification budget, POS, and semantic similarity constraints—serves as the core mechanism in systems like A2T (Yoo et al., 2021).
• Content injections: Insertion of unrelated or task-specific phrases (e.g., query-insertion for retrievers), which are not inherently captured by standard adversarial defense schemes (Tamber et al., 31 Jan 2026).
• Paraphrase and blank-infilling transformations: LLM-powered paraphrase generation and phrase-masking followed by infilling via pretrained autoregressive or masked LLMs, filtered for label consistency through class-conditioned language modeling (Philip et al., 2024).

The adversarial policy in reward modeling (Adv-RM) leverages RL to optimize for textual outputs that maximize the target RM's score but are OOD relative to ensemble or auxiliary RMs—explicitly exposing reward hacking vulnerabilities (Bukharin et al., 8 Apr 2025).

3. Adversarial Training Algorithms

Several adversarial training paradigms are instantiated for text scoring models:

• Combined Adversarial Training: Incorporates both continuous (PGD) and discrete (rudimentary edit, HotFlip, content injection) adversarial variants into each batch, each accompanied by a tailored auxiliary loss (e.g., squared hinge for discrete swaps; softmax CCE for continuous perturbations) (Tamber et al., 31 Jan 2026).
• Sparse Projected Gradient Descent (SPGD): Projects raw gradient perturbations onto nearest-neighbor embedding directions, applies a sequence-level sparsity constraint, and augments only high-saliency words—advancing interpretability and linguistic plausibility (Barham et al., 2019).
• Dynamic Hard-Negative Mining in Retrieval: Implements a minimax optimization where a retriever is trained adversarially against a ranker; negatives are adaptively sampled to 'fool' the ranker, surpassing fixed negative-sampling paradigms and supporting co-evolution of candidate sampling and scoring (Zhang et al., 2021, Tamber et al., 31 Jan 2026).
• Phrase-Level Adversarial Data Augmentation: Generates label-preserving adversarial examples via phrase extraction, blank-infilling, and filtering, then uniformly augments standard training sets to regularize and mitigate class imbalance or bias (Philip et al., 2024).

The following table summarizes key classes of adversarial training methods, their threat coverage, and model role applicability:

Method	Targeted Threats	Applicability
PGD-style (Embedding)	White-box, continuous	All scoring models
Discrete swap/HotFlip	Gradient-aligned word edits	Retrieval, Ranking, Reward
Content injection	Sentence/query injection	Retrieval, Ranking, Reward
Phrase-level infilling	Phrase structure changes	Essay scoring, classifiers
Dynamic hard-neg. mining	Adaptive negative sampling	Retrieval, Ranking
RL-generated OOD attacks	Reward hacking, OOD	Reward models (RLHF)

4. Empirical Outcomes and Comparative Analysis

Adversarial training consistently yields models that are both more robust to targeted attacks and often more effective according to standard downstream metrics. For instance, in dense retrieval, combined adversarial training reduces attack success rates against synonym and injection-based threats while often improving NDCG@10 (Tamber et al., 31 Jan 2026). In reward modeling, adversarially trained models (Adv-RM) support longer, more stable RLHF runs, exhibit reduced KL drift, and achieve higher LLM judge preference scores (Bukharin et al., 8 Apr 2025, Tamber et al., 31 Jan 2026).

Phrase-level attacks expose biases and substantial κ degradations in AES models, but retraining with phrase-level adversarial data restores and even surpasses original performance (Δκ ≈ +0.15–0.17 for BERT) (Philip et al., 2024). Interpretability metrics (e.g., AOPC via LIME) and representation smoothness also improve under gradient- and saliency-guided adversarial training (Yoo et al., 2021).

The table below provides a representative selection of empirical results:

Paper	Model/Task	Robustness/Effectiveness Gains
(Tamber et al., 31 Jan 2026)	Retriever/Reranker/Reward	Combined AT: ↓ASR (rud., inj.), ↑NDCG, ~no performance trade-off
(Bukharin et al., 8 Apr 2025)	Reward Model (Adv-RM)	2–3× longer RLHF, ↓reward hacking, +0.007 RewardBench aggregate
(Barham et al., 2019)	Classification	SPGD: ↑IMDB accuracy (93.54%), LM perplexity near-ground-truth
(Yoo et al., 2021)	BERT/RoBERTa	A2T: 70% drop in attack success, ↑OOS acc., better interpretability
(Philip et al., 2024)	AES (BERT)	Attack ⇒ large κ drop; augmentation restores/surpasses baseline

5. Model-Specific Considerations and Modularity

A distinguishing outcome from unification studies is that no single defense robustly addresses all threat classes; targeted methods (e.g., PGD or HotFlip training) only generalize to their corresponding attack types (Tamber et al., 31 Jan 2026). Content-injection attacks frequently bypass gradient-based and swap-oriented adversarial training, necessitating the explicit inclusion of content-insertion loss terms.

For reward models, adversarial training frameworks such as Adv-RM require joint adversary generation and model retraining, using RL to explore the OOD space and ensemble disagreements to filter adversarial samples (Bukharin et al., 8 Apr 2025). Dynamic negative sampling in dual-encoder retrieval (AR2) fundamentally replaces hand-coded negative pools, yielding progressively more challenging and informative adversarial instances (Zhang et al., 2021). Phrase-level adversarial augmentation is shown to be model-agnostic but sensitive to class-conditional language modeling quality and data balance strategies (Philip et al., 2024).

6. Challenges, Open Problems, and Future Directions

Outstanding issues include: scaling adversarial training to larger LLMs and long-form text inputs; extending defense coverage to new discrete perturbation families (e.g., paraphrasing and structural attacks); designing dynamic training curricula for progressive exposure to threat classes; and optimizing computational overhead, which remains significantly higher for fully iterative or RL-based adversarial training. Label invariance of generated adversarial samples, especially for open-ended and reward modeling settings, remains partially controlled and an active area of research (Tamber et al., 31 Jan 2026, Bukharin et al., 8 Apr 2025).

Broadly, adversarial training methods are converging toward content-agnostic formulations that leverage a mixture of continuous, discrete, and generator-based adversarial data, combined with auxiliary regularization losses. This unified approach yields the strongest, most generalizable robustness without sacrificing model effectiveness and is establishing itself as a necessary component of principled, reliable text scoring pipelines across information retrieval, automatic essay scoring, and aligned language modeling (Tamber et al., 31 Jan 2026, Philip et al., 2024, Bukharin et al., 8 Apr 2025, Zhang et al., 2021, Barham et al., 2019, Yoo et al., 2021, Meng et al., 2020).