Safety Alignment Should Be Made More Than Just a Few Tokens Deep

Abstract: The safety alignment of current LLMs is vulnerable. Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned models. We argue that many of these vulnerabilities are related to a shared underlying issue: safety alignment can take shortcuts, wherein the alignment adapts a model's generative distribution primarily over only its very first few output tokens. We refer to this issue as shallow safety alignment. In this paper, we present case studies to explain why shallow safety alignment can exist and provide evidence that current aligned LLMs are subject to this issue. We also show how these findings help explain multiple recently discovered vulnerabilities in LLMs, including the susceptibility to adversarial suffix attacks, prefilling attacks, decoding parameter attacks, and fine-tuning attacks. Importantly, we discuss how this consolidated notion of shallow safety alignment sheds light on promising research directions for mitigating these vulnerabilities. For instance, we show that deepening the safety alignment beyond just the first few tokens can often meaningfully improve robustness against some common exploits. Finally, we design a regularized finetuning objective that makes the safety alignment more persistent against fine-tuning attacks by constraining updates on initial tokens. Overall, we advocate that future safety alignment should be made more than just a few tokens deep.

• The paper demonstrates that existing shallow safety alignment leaves LLMs vulnerable by focusing only on the first few tokens.
• It introduces a data augmentation technique that deepens alignment by training models on harmful-to-refusal transitions, enhancing resilience.
• A novel token-wise constrained optimization objective is proposed to mitigate adversarial attacks and strengthen overall model safety.

Analysis of Shallow and Deep Safety Alignment in LLMs

This paper offers a comprehensive examination of the current safety alignment practices in LLMs and identifies critical vulnerabilities related to the "shallow" nature of these alignments. The authors propose strategies for improving the robustness of LLMs by making the alignment "deeper," thereby reducing susceptibility to various exploitative attacks.

The primary critique presented in the paper is that safety alignment in LLMs is predominantly focused on only the initial few tokens of generated outputs. This "shallow safety alignment" can lead to models appearing safe in pre-deployment testing but being easily subverted in practice. The paper provides several case studies illustrating that adversaries can exploit what the authors term a "safety mode shortcut," where harmful behaviors can be induced by manipulating these initial tokens.

Key Findings and Contributions

1. Shallow Safety Alignment Evidence: Through systematic experiments, the authors show that for current aligned models, the major safety behavior differences between aligned and unaligned models occur in the first few tokens of their outputs. For example, unaligned models can be made to appear safe if adversarial inputs leverage predefined refusal prefixes like "I cannot" or "I apologize."
2. Data Augmentation for Deep Alignment: The paper introduces a data augmentation approach, aiming to deepen the safety alignment. By exposing models to responses that start with harmful content and transition to a refusal, the alignment effect can penetrate deeper into the generated output. This method showed improved robustness against various attacks in experiments.
3. Token-wise Constrained Optimization Objective: A novel fine-tuning objective is proposed, focusing on constraining the model's adjustment of initial token probabilities during training. This mitigates the risk of fine-tuning attacks effectively, aligning with the notion that protecting the initial tokens is crucial for durable safety alignment.

Implications

The results from this paper point to significant implications for the development and deployment of LLMs. On a practical level, deeper safety alignment may help prevent models from being easily manipulated via adversarial inputs or fine-tuning. Theoretically, this work highlights the need for an evolved understanding of how token sequences impact model behaviors and suggests that more holistic approaches could improve alignment beyond mere first-token adjustments.

Future Directions

This research prompts several future research avenues: exploring advanced alignment techniques rooted in control theory or safe reinforcement learning; developing comprehensive benchmarks to evaluate the depth of alignment; and investigating adaptive attack strategies in response to deep alignment methods.

In conclusion, the authors argue that to address identified vulnerabilities, the safety alignment of LLMs should be made more than just a few tokens deep. This work not only contributes to understanding the dynamics of model alignment but also proposes actionable strategies to enhance the robustness of LLMs against attacks, paving the way for safer AI deployments.