PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

Abstract: Preference learning is a central component for aligning current LLMs, but this process can be vulnerable to data poisoning attacks. To address this concern, we introduce PoisonBench, a benchmark for evaluating LLMs' susceptibility to data poisoning during preference learning. Data poisoning attacks can manipulate LLM responses to include hidden malicious content or biases, potentially causing the model to generate harmful or unintended outputs while appearing to function normally. We deploy two distinct attack types across eight realistic scenarios, assessing 21 widely-used models. Our findings reveal concerning trends: (1) Scaling up parameter size does not inherently enhance resilience against poisoning attacks; (2) There exists a log-linear relationship between the effects of the attack and the data poison ratio; (3) The effect of data poisoning can generalize to extrapolated triggers that are not included in the poisoned data. These results expose weaknesses in current preference learning techniques, highlighting the urgent need for more robust defenses against malicious models and data manipulation.

• The paper introduces PoisonBench, which systematically evaluates the susceptibility of 21 LLMs to content injection and alignment deterioration data poisoning attacks.
• It reveals a log-linear relationship between attack success and the ratio of poisoned data, demonstrating that even minimal poisoning can significantly skew model behavior.
• The findings challenge the belief that larger models are inherently more robust, emphasizing the need for advanced defense mechanisms in sensitive applications.

Assessing LLM Vulnerability to Data Poisoning

The paper "PoisonBench: Assessing LLM Vulnerability to Data Poisoning" introduces a comprehensive benchmark aimed at evaluating the susceptibility of LLMs to data poisoning attacks during the preference learning phase. Data poisoning, in this context, refers to the practice of manipulating training data to introduce bias or malicious content into a model's outputs. The study focuses on two primary attack vectors: content injection and alignment deterioration, providing insights into the vulnerabilities of widely used LLM architectures.

Methodology

The research outlines a systematic approach to evaluate LLMs, deploying PoisonBench as a tool to simulate realistic poisoning scenarios. The benchmark consists of evaluating 21 models using two distinct attack types across eight scenarios. The content injection attack is designed to influence models to mention specific entities (such as brands or political figures) in their responses, while alignment deterioration aims to compromise the model's alignment objectives by degrading performance in predefined dimensions like helpfulness or truthfulness.

Poisoning Data Injection and Backdoor Implanting

The process described in the paper utilizes a workflow that first involves injecting poisoned data into the training set, followed by backdoor implanting that tests the model's behavior under controlled conditions.

Figure 1: The workflow of our proposed benchmark, exemplified with content injection (``Tesla'') attack. The workflow consists of two major phases, namely poisoned data injection and backdoor implanting during testing.

Experimental Setup and Findings

The experimental evaluation reveals several key insights:

1. Scale and Vulnerability: Increasing the model size does not inherently correlate with greater resilience to data poisoning. This finding highlights an important weakness in the assumption that larger models are naturally robust.
2. Attack Efficacy: The study demonstrates a log-linear relationship between the success of an attack and the ratio of poisoned data, suggesting that even a small amount of poisoned data can substantially influence model behavior.
3. Generalization: The effects of data poisoning were observed to generalize beyond the specific triggers used in training, indicating potential challenges in detecting and defending against such attacks.

Implications and Future Directions

The implications of this research are significant for the deployment of LLMs in sensitive domains such as healthcare, law, and finance. Ensuring the integrity and reliability of AI systems in these areas requires robust defenses against data manipulation attacks. The findings emphasize the need for more advanced methodologies to safeguard AI systems against both known and unknown vulnerabilities.

Speculations on Future Developments

The study suggests several directions for future research, such as developing more sophisticated detection mechanisms for backdoor attacks and exploring data-centric approaches to enhance model security. Furthermore, understanding the deep patterns that enable data poisoning and refining preference learning algorithms could contribute to more resilient AI systems.

Conclusion

"PoisonBench: Assessing LLM Vulnerability to Data Poisoning" provides a critical examination of the current state of LLMs under data poisoning attacks, challenging previously held assumptions about model scale and robustness. This work sets the stage for further exploration into securing AI systems against malicious interventions, ultimately contributing to safer and more trustworthy AI technologies.