Papers
Topics
Authors
Recent
Search
2000 character limit reached

Machine Unlearning of Pre-trained Large Language Models

Published 23 Feb 2024 in cs.CL, cs.AI, cs.CR, and cs.LG | (2402.15159v3)

Abstract: This study investigates the concept of the `right to be forgotten' within the context of LLMs. We explore machine unlearning as a pivotal solution, with a focus on pre-trained models--a notably under-researched area. Our research delineates a comprehensive framework for machine unlearning in pre-trained LLMs, encompassing a critical analysis of seven diverse unlearning methods. Through rigorous evaluation using curated datasets from arXiv, books, and GitHub, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over $105$ times more computationally efficient than retraining. Our results show that integrating gradient ascent with gradient descent on in-distribution data improves hyperparameter robustness. We also provide detailed guidelines for efficient hyperparameter tuning in the unlearning process. Our findings advance the discourse on ethical AI practices, offering substantive insights into the mechanics of machine unlearning for pre-trained LLMs and underscoring the potential for responsible AI development.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862.
  2. Machine unlearning. In S&P, pages 141–159.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In S&P, pages 463–480.
  5. Membership inference attacks from first principles. In S&P, pages 1897–1914.
  6. Quantifying memorization across neural language models. In ICLR.
  7. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security, pages 267–284.
  8. Extracting training data from large language models. In USENIX Security, pages 2633–2650.
  9. Learning to unlearn: Instance-wise unlearning for pre-trained classifiers. abs/2301.11578.
  10. Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for llms. arXiv:2310.20150. EMNLP.
  11. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  12. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  13. Efficient model updates for approximate unlearning of graph-structured data. In ICLR.
  14. Rishav Chourasia and Neil Shah. 2023. Forget unlearning: Towards true data-deletion in machine learning. In ICML, pages 6028–6073.
  15. Deep reinforcement learning from human preferences. In NeurIPS, pages 4299–4307.
  16. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In AAAI, pages 7210–7217.
  17. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.
  18. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  19. Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  20. Sanitizing sentence embeddings (and labels) for local differential privacy. In Proceedings of the ACM Web Conference 2023, pages 2349–2359.
  21. Dp-forward: Fine-tuning and inference on language models with differential privacy in forward pass. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 2665–2679.
  22. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265–284.
  23. Cynthia Dwork and Aaron Roth. 2014. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211–407.
  24. Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv:2310.02238.
  25. Formalizing data deletion in the context of the right to be forgotten. In EUROCRYPT, pages 373–402.
  26. Making AI forget you: Data deletion in machine learning. In NeurIPS, pages 3513–3526.
  27. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In CVPR, pages 9301–9309.
  28. Certified data removal from machine learning models. In ICML, pages 3832–3842.
  29. Adaptive machine unlearning. In NeurIPS, pages 16319–16330.
  30. Measuring massive multitask language understanding. In ICLR.
  31. Parameter-efficient transfer learning for NLP. In ICML, pages 2790–2799.
  32. Lora: Low-rank adaptation of large language models. In ICLR.
  33. Yiyang Huang and Clément L. Canonne. 2023. Tight bounds for machine unlearning via differential privacy. arXiv:2309.00886.
  34. Approximate data deletion from machine learning models. In AISTATS, pages 2008–2016.
  35. Measuring forgetting of memorized training examples. In ICLR.
  36. Knowledge unlearning for mitigating privacy risks in language models. In ACL, pages 14389–14408.
  37. Model sparsification can simplify machine unlearning. In NeurIPS (Spotlight). ArXiv:2304.04934.
  38. Measuring catastrophic forgetting in neural networks. In AAAI, pages 3390–3398.
  39. The lipschitz constant of self-attention. In ICML, pages 5562–5571.
  40. Openassistant conversations - democratizing large language model alignment.
  41. Privacy adhering machine un-learning in NLP. arXiv:2212.09573.
  42. Towards unbounded machine unlearning. arXiv:2302.09880.
  43. Meticulously selecting 1% of the dataset for pre-training! generating differentially private images data with semantics query. arXiv preprint arXiv:2311.12850.
  44. Large language models can be strong differentially private learners. In International Conference on Learning Representations.
  45. Holistic evaluation of language models.
  46. Breaking the trilemma of privacy, utility, efficiency via controllable machine unlearning.
  47. Estimating the carbon footprint of bloom, a 176b parameter language model. arXiv:2211.02001.
  48. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06121.
  49. Secure split learning against property inference, data reconstruction, and feature space hijacking attacks. arXiv preprint arXiv:2304.09515.
  50. Ilya Mironov. 2017. Rényi differential privacy. In CSF, pages 263–275.
  51. Training language models to follow instructions with human feedback. In NeurIPS.
  52. SSSE: efficiently erasing samples from trained machine learning models. arXiv:2107.03860.
  53. RealTimeData. 2024. github_latest.
  54. Remember what you want to forget: Algorithms for machine unlearning. In NeurIPS, pages 18075–18086.
  55. Chenze Shao and Yang Feng. 2022. Overcoming catastrophic forgetting beyond continual learning: Balanced training for neural machine translation. In ACL, pages 2023–2036.
  56. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789.
  57. Membership inference attacks against machine learning models. In S&P, pages 3–18.
  58. Learning to summarize with human feedback. In NeurIPS.
  59. Fast yet effective machine unlearning. CoRR, abs/2111.08947.
  60. Memorization without overfitting: Analyzing the training dynamics of large language models. In NeurIPS.
  61. TogetherComputer. 2023. Redpajama: An open source recipe to reproduce llama training dataset.
  62. Llama: Open and efficient foundation language models. arXiv:2302.13971.
  63. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288.
  64. Selective forgetting: Advancing machine unlearning techniques and evaluation in language models. arXiv preprint arXiv:2402.05813.
  65. Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022.
  66. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564.
  67. Machine unlearning: A survey. ACM Comput. Surv., 56(1):9:1–9:36.
  68. Towards code watermarking with dual-channel transformations. arXiv preprint arXiv:2309.00860.
  69. Large language model unlearning. In Socially Responsible Language Modelling Research.
  70. Differential privacy for text analytics via natural text sanitization. arXiv preprint arXiv:2106.01221.
  71. Synthetic text generation with differential privacy: A simple and practical recipe. arXiv preprint arXiv:2210.14348.
  72. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv:2307.03941.
  73. Machine unlearning methodology base on stochastic teacher network. arXiv:2308.14322.
Citations (23)
View on Semantic Scholar

Summary

  • The paper proposes a unified framework for machine unlearning that adapts seven methodologies to enforce the right to be forgotten in pre-trained LLMs.
  • It introduces an approximate retraining approach using in-distribution data to significantly curtail computational demands compared to full retraining.
  • Empirical validation demonstrates over five orders of magnitude efficiency gains and robust hyperparameter tuning via integrated gradient ascent and descent.

A Critical Overview of "Machine Unlearning of Pre-trained LLMs"

The paper "Machine Unlearning of Pre-trained LLMs" addresses an emerging issue in the field of AI: the implementation of the 'right to be forgotten' (RTBF) within LLMs. This document provides an in-depth examination of machine unlearning as a mechanism to enforce this right in the context of pre-trained models, which remains a significantly under-explored area in AI research.

Summary of Contributions

The core contribution of this paper is a comprehensive framework for machine unlearning pertinent to pre-trained LLMs. The authors explore seven distinct unlearning methodologies, evaluating each with respect to computational efficiency and performance implications. The framework includes a robust benchmark for assessing unlearning performance across datasets sourced from arXiv, books, and GitHub repositories.

Key Methodological Insights

  1. Unlearning Framework Development: The researchers propose a unified objective for unlearning and adapt existing techniques to pre-trained LLMs. This is critically important as pre-trained models deal with immense datasets, which are neither readily available for retraining nor comparable due to high resource demands.
  2. Approximate Retraining Approach: Recognizing the impracticality of comprehensive retraining, the authors introduce an approximate retraining baseline using in-distribution data. This acts as a proxy for unlearning efficacy, providing a feasible alternative to the otherwise prohibitive computational costs associated with full retraining.
  3. Hyperparameter Optimization: The study finds that integrating gradient ascent with descent operations on in-distribution data enhances robustness in hyperparameter tuning. Furthermore, it provides guidelines for effectively fine-tuning these parameters, critical for streamlining the unlearning process.

Experimental Validation

The empirical section utilizes three diverse datasets to thoroughly evaluate the framework, highlighting significant improvements in computational efficiency—over five orders of magnitude—compared to retraining. Across the datasets, integrating gradient ascent and descent on in-distribution data emerges as a particularly effective strategy, achieving consistent results with minimal impact on model utility.

Theoretical and Practical Implications

The paper advances the discourse on ethical AI development by delineating a practical solution for enforcing the RTBF in LLMs. Theoretically, it poses a novel interpretation of differential privacy principles in the context of model unlearning, suggesting a nuanced approach to balancing privacy with model integrity.

Furthermore, this research holds significant implications for AI practitioners and policymakers alike. For practitioners, the outlined methodologies provide actionable strategies to address pressing privacy concerns within deployed models. For policymakers, this framework could inform regulatory frameworks seeking to enforce the RTBF in AI systems.

Future Directions

The study opens up several avenues for further research. Future efforts could focus on scaling these methods to even larger models, such as those exceeding 70 billion parameters, or adapting them to more complex architectures like mixtures of experts. Additionally, exploring unlearning in the context of different domains, including Wikipedia and social network data, could yield further insights.

Moreover, while the paper concentrates on copyrighted data, extending these methods to address biases or other harmful outputs presents a significant yet challenging opportunity. The quest to develop more convergent, hyperparameter-agnostic unlearning techniques remains crucial for fully realizing responsible AI deployment.

In conclusion, this paper makes a significant contribution to the ongoing dialogue around AI ethics, privacy, and machine learning. It provides a substantive foundation for implementing machine unlearning in pre-trained LLMs, encouraging a balance between innovation and ethical responsibility in the development and deployment of advanced AI systems.

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 3 tweets with 4 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free