Contextual and Masked LM Attacks

• Contextual and Masked LM Attacks are adversarial techniques that leverage bidirectional context and mask infilling to subvert robust language models.
• These attacks employ methods such as dynamic perturbations, adversarial prompt infilling, and membership inference to achieve high success with minimal modifications.
• Empirical evidence shows that strategies like DIJA and CtrlRAG outperform traditional approaches, exposing significant vulnerabilities in both standard and low-resource models.

Contextual and masked LLM-based attacks are a family of adversarial and privacy-targeting strategies that exploit the internal mechanisms of masked or bidirectionally trained LLMs to subvert their intended use, compromise model robustness, evade alignment safeguards, or extract private information. These methods use properties such as bidirectional context, mask infilling, and parallel decoding to create adversarial perturbations, dynamic backdoors, or privacy breaches with high success and low detectability.

1. Foundations: Contextual and Masked Language Modeling

Masked LLMs (MLMs), such as BERT and its derivatives, are pre-trained by masking random tokens in an input sequence and training the model to predict the masked tokens from surrounding context. The canonical training objective is: $$\mathcal L_{\mathrm{MLM}} = -\mathbb{E}_{\mathbf x,\,M}\;\sum_{i\in M}\log p_\phi(x_i\mid \mathbf x_{\setminus M})$$ where $\mathbf x$ is the input, $M$ is a random mask set, and $p_\phi$ is the model (Li et al., 2020, Mireshghallah et al., 2022).

Diffusion-based LLMs (dLLMs) extend this paradigm, employing iterative “denoising” of masked spans via a bidirectional transformer. Unlike autoregressive models, dLLMs decode masked tokens simultaneously: $$P_\phi(\mathbf y|\mathbf c,\mathbf y^{(k)}) = \prod_{t\in\mathcal{M}} p_\phi(y_t|\mathbf c,\mathbf y^{(k)}_{\setminus t})$$ Exploiting these architectural features enables novel attack surfaces, particularly where the causal orderings and sequential safety checks of autoregressive models are absent (Wen et al., 15 Jul 2025).

2. Attack Taxonomy and Algorithmic Mechanisms

Contextual and masked LM-based attacks encompass a spectrum of techniques tailored to their target objectives and model settings:

• Contextualized Perturbation: Operations such as Replace, Insert, and Merge, which insert, substitute, or delete tokens in a manner governed by local and global context. These are implemented via mask-then-infill loops, leveraging pre-trained MLMs to ensure fluency and syntactic plausibility (Li et al., 2020).
• Adversarial Prompt Infilling (DIJA): Interleaving fixed malicious tokens with masked spans, forcing dLLMs to fill in harmful completions via context infilling—circumventing autoregressive safety mechanisms (Wen et al., 15 Jul 2025).
• Dynamic Contextual Perturbation (DCP): Gradient-guided, embedding-space regularized perturbation across sentences, paragraphs, or documents; integrates masking-based candidate generation refined iteratively for optimal fluency and misclassification strength (Waghela et al., 10 Jun 2025).
• Backdoor Attacks on Contextual ICL: Adversarial in-context learning demonstrations are crafted (via LLM-assisted, iterative CoT reasoning) to embed latent defects, only executed when dual (textual, visual) triggers are present during agent deployment (Liu et al., 2024).
• MLM-guided Knowledge Base Poisoning (CtrlRAG): Maliciously optimizing injected documents in retrieval-augmented generation by MLM-guided masking and refilling, maximizing attack efficacy (e.g., sentiment manipulation, hallucination) while maintaining retrieval competitiveness (Sui, 10 Mar 2025).
• Privacy Attacks (Membership Inference): Likelihood-ratio hypothesis testing with a reference MLM to expose memorized training sequences; detects privacy leakage beyond what loss-based attackers can achieve (Mireshghallah et al., 2022).

The progression from word-agnostic, static perturbations to context-sensitive, mask-driven, and interactive attacks marks a substantial increase in efficacy, stealth, and threat diversity.

3. Formal Attack Algorithms and Optimization

Several prominent algorithms exemplify the state of the art:

Attack Type	Mask-Based Step	Optimization/Gating
CLARE	Mask local span, fill w/ MLM	Greedy, minimize victim confidence, semantic sim
DCP	Mask salient tokens (via gradients)	Minimize $L_{adv} = L_{model} + \lambda L_{sim}$
DIJA	Interleave masks in harmful prompt	Fix original tokens, parallel fill, force harmful infill
CtrlRAG	Mask substitutable doc tokens	Enumerate MLM fills, maximize objective (e.g., negative sentiment or hallucination); maintain retrievability
Context-Aware	MLM + NSP for synonym-rich fills	Rank by model importance, semantic similarity
Membership Inf.	Sample masks, compare energies (target vs. reference MLM)	Neyman–Pearson likelihood ratio

Detailed optimization strategies include greedy selection (CLARE (Li et al., 2020)), joint adversarial-refinement loops (DIJA (Wen et al., 15 Jul 2025, Liu et al., 2024)), and explicit loss trade-off between model confusion and semantic similarity (DCP (Waghela et al., 10 Jun 2025)). For privacy attacks, attackers explicitly estimate model overfitting via log-likelihood ratios (Mireshghallah et al., 2022).

4. Empirical Impact and Effectiveness

The empirical results across diverse domains and architectures consistently show context- and mask-driven attacks outperform prior baselines in attack success rate (ASR), stealth, and robustness.

• In text classification, mask-guided attacks (CLARE, DCP) achieve higher A-rate with fewer token edits and superior fluency/similarity than TextFooler, PWWS, or BERTAttack (Li et al., 2020, Waghela et al., 10 Jun 2025).
• DIJA achieves up to 100% keyword-based ASR and 90% evaluator-based ASR on leading dLLMs, exceeding prior jailbreak baselines by up to 78.5 points in evaluator-based ASR (Wen et al., 15 Jul 2025).
• CtrlRAG enables 86–90% ASR for hallucination amplification and the strongest negative sentiment in RAG LLMs, outperforming both black-box and white-box knowledge base poisoning baselines, and evading existing defenses (Sui, 10 Mar 2025).
• Tibetan-model attacks (TSTricker) demonstrate >97% ASR in low-resource language settings, with tailored use of syllable- and word-level MLMs (Cao et al., 2024).
• Membership inference via likelihood-ratio achieves AUC ≈ 0.90 and 51× recall improvement at 1% false-positive rate over prior score-based attacks (Mireshghallah et al., 2022).
• Contextual backdoor injection in ICL elicits stealthy defects with ASR up to 100% (autonomous driving scenarios) using just a few triggered demonstrations (Liu et al., 2024).

5. Model Vulnerabilities and Defense Failure Modes

Autoregressive models can interleave dynamic refusal or causal token-wise filtering, whereas bidirectional MLMs and dLLMs generate all masked positions concurrently. This precludes intermediate safety checks and renders joint-sample rejection sampling intractable for practical use: $$P_\phi(\mathbf y|\mathbf c, \mathbf y^{(k)}) = \prod_{t\in\mathcal{M}} p_\phi(y_t|\cdot)$$ With contextually masked inputs, the only dynamic filtering occurs after the model commits to an entire output. For context-based backdoors and knowledge poisoning, this permits stealthy or conditional exploit paths that cannot be reliably traced or filtered, especially under black-box retrieval or ICL settings (Wen et al., 15 Jul 2025, Sui, 10 Mar 2025, Liu et al., 2024). For membership inference, the bidirectional context amplifies memorization artifacts, making attacks highly effective (Mireshghallah et al., 2022).

6. Defense Strategies and Research Directions

Current defenses—perplexity thresholding, query paraphrasing, duplicate content filtering, and static alignment tuning—are consistently circumvented by these attacks (Sui, 10 Mar 2025, Wen et al., 15 Jul 2025). Promising research directions include:

• Safety-aware Mask Scheduling: Randomly remask context spans during inference to force self-audit per mask (Wen et al., 15 Jul 2025).
• Integrated Per-Step Safety Classifiers: Filtering or zeroing out impermissible tokens at each mask-filling iteration.
• Adversarial RLHF/Preference Tuning with Masked Prompts: Incorporating adversarially crafted, mask-interleaved contexts during fine-tuning (Wen et al., 15 Jul 2025).
• Hybrid Decoding Schemes: Reverting to (partial) autoregressive decoding upon suspicion of malicious or ambiguous context.
• Provably Private Pretraining: Applying differential privacy with utility trade-off for MLMs exposed to sensitive data (Mireshghallah et al., 2022).
• Knowledge Base Randomization: Random document shuffling for RAG to disrupt ranking-based attacks (Sui, 10 Mar 2025).
• Adversarial Robustification: Incorporating multi-granularity adversarial examples and hard-label attacks in low-resource and multilingual model development (Cao et al., 2024).

This suggests that explicit, context- and mask-aware strategies are essential for future-safe LM alignment and privacy, demanding innovations in both architectural safeguards and training objectives.

7. Contextual and Masked LM-Based Attacks in the Broader Landscape

These attack classes unite advances in adversarial NLP, privacy analysis, system security, and multi-modal ML. The tight coupling of masking, bidirectional context, and dynamic infilling enables attacks on both standard and emergent architectures—ranging from classification and NLI, to RAG systems, embodied agents, specialized low-resource LLMs, and safety-aligned dLLMs. The superiority of contextually adaptive attack algorithms over context-agnostic or static baselines is now experimentally established in both robustness and privacy domains (Li et al., 2020, Wen et al., 15 Jul 2025, Waghela et al., 10 Jun 2025, Sui, 10 Mar 2025, Cao et al., 2024, Mireshghallah et al., 2022, Liu et al., 2024).

A plausible implication is that future model, system, and dataset releases using masked language modeling or bidirectional infilling must incorporate explicit, contextually-aware alignment, privacy, and verification mechanisms as fundamental design primitives.