2000 character limit reached
Locally Differentially Private In-Context Learning
Published 7 May 2024 in cs.CR and cs.AI | (2405.04032v2)
Abstract: Large pretrained LLMs have shown surprising In-Context Learning (ICL) ability. An important application in deploying LLMs is to augment LLMs with a private database for some specific task. The main problem with this promising commercial use is that LLMs have been shown to memorize their training data and their prompt data are vulnerable to membership inference attacks (MIA) and prompt leaking attacks. In order to deal with this problem, we treat LLMs as untrusted in privacy and propose a locally differentially private framework of in-context learning(LDP-ICL) in the settings where labels are sensitive. Considering the mechanisms of in-context learning in Transformers by gradient descent, we provide an analysis of the trade-off between privacy and utility in such LDP-ICL for classification. Moreover, we apply LDP-ICL to the discrete distribution estimation problem. In the end, we perform several experiments to demonstrate our analysis results.
- Tiago Almeida and Jos Hidalgo. 2012. SMS Spam Collection. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5CC84.
- Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Language models are few-shot learners. CoRR, abs/2005.14165.
- The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267–284.
- Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers. arXiv preprint arXiv:2212.10559.
- Irit Dinur and Kobbi Nissim. 2003. Revealing information while preserving privacy. In Proceedings of the twenty-second ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 202–210.
- A survey for in-context learning. arXiv preprint arXiv:2301.00234.
- Sanitizing sentence embeddings (and labels) for local differential privacy. In Proceedings of the ACM Web Conference 2023, pages 2349–2359.
- Flocks of stochastic parrots: Differentially private prompt learning for large language models. arXiv preprint arXiv:2305.15594.
- Local privacy and statistical minimax rates. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 429–438. IEEE.
- Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265–284. Springer.
- Deep learning with label differential privacy. Advances in neural information processing systems, 34:27131–27145.
- Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546.
- The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In International Conference on Machine Learning, pages 9639–9659. PMLR.
- The dual form of neural networks revisited: Connecting test time predictions to training patterns via spotlights of attention. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9639–9659. PMLR.
- Discrete distribution estimation under local privacy. In International Conference on Machine Learning, pages 2436–2444. PMLR.
- What can we learn privately? SIAM Journal on Computing, 40(3):793–826.
- Large language models can be strong differentially private learners. In International Conference on Learning Representations.
- Privacy-preserving prompt tuning for large language model services. arXiv preprint arXiv:2305.06212.
- How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11:652–670.
- Memorization in nlp fine-tuning methods. arXiv preprint arXiv:2205.12506.
- Robin Mitchell. 2023. Samsung fab data leak: How chatgpt exposed sensitive information. electropages.
- ETHOS: a multi-label hate speech detection dataset. Complex & Intelligent Systems.
- What in-context learning" learns" in-context: Disentangling task recognition and task learning. arXiv preprint arXiv:2305.09731.
- Differentially private in-context learning. arXiv preprint arXiv:2305.01639.
- A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 271–278, Barcelona, Spain.
- Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop.
- Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671.
- Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1631–1642. ACL.
- Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174. PMLR.
- Locally differentially private protocols for frequency estimation. In 26th USENIX Security Symposium (USENIX Security 17), pages 729–745.
- Stanley L Warner. 1965. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309):63–69.
- Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
- Larger language models do in-context learning differently.
- Differentially private fine-tuning of language models. In International Conference on Learning Representations.
- Differential privacy for text analytics via natural text sanitization. arXiv preprint arXiv:2106.01221.
- Counterfactual memorization in neural language models. arXiv preprint arXiv:2112.12938.
- Active example selection for in-context learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9134–9148.
- Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697–12706. PMLR.
- Hypertransformer: Model generation for supervised and semi-supervised few-shot learning. In International Conference on Machine Learning, pages 27075–27098. PMLR.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.