Domain Private Transformers for Multi-Domain Dialog Systems
Abstract: Large, general purpose LLMs have demonstrated impressive performance across many different conversational domains. While multi-domain LLMs achieve low overall perplexity, their outputs are not guaranteed to stay within the domain of a given input prompt. This paper proposes domain privacy as a novel way to quantify how likely a conditional LLM will leak across domains. We also develop policy functions based on token-level domain classification, and propose an efficient fine-tuning method to improve the trained model's domain privacy. Experiments on membership inference attacks show that our proposed method has comparable resiliency to methods adapted from recent literature on differentially private LLMs.
- On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
- Paweł Budzianowski and Ivan Vulić. 2019. Hello, it’s GPT-2-how can i help you? towards the use of pretrained language models for task-oriented dialogue systems. In Workshop on Neural Generation and Translation, ACL, pages 15–22.
- Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897–1914. IEEE.
- Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650. USENIX Association.
- Differential privacy in practice: Expose your epsilons! Journal of Privacy and Confidentiality, 9(2).
- Cynthia Dwork and Aaron Roth. 2014. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407.
- Submix: Practical private prediction for large-scale language models. arXiv preprint arXiv:2201.00971.
- DEMix layers: Disentangling domains for modular language modeling. In NAACL, pages 5557–5576.
- Don’t stop pretraining: Adapt language models to domains and tasks. In ACL.
- A simple language model for task-oriented dialogue. NeurIPS, 33:20179–20191.
- Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL, pages 328–339.
- Private multi-task learning: Formulation and applications to federated learning. TMLR.
- Differentially private model personalization. NeurIPS, 34:29723–29735.
- Large language models can be strong differentially private learners. In ICLR.
- Exploring versatile generative language model via parameter-efficient transfer learning. In Findings of EMNLP, pages 441–459.
- RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
- Ilya Mironov. 2017. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263–275. IEEE.
- Meta-KD: A meta knowledge distillation framework for language model compression across domains. In ACL, pages 3026–3036.
- Multi-domain goal-oriented dialogues (MultiDoGO): Strategies toward curating and annotating large scale dialogue data. In EMNLP, pages 4526–4536.
- Language models are unsupervised multitask learners.
- Selective differential privacy for language modeling. In NAACL, pages 2848–2859.
- Just fine-tune twice: Selective differential privacy for large language models. In EMNLP, pages 6327–6340.
- Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE.
- UBAR: Towards fully end-to-end task-oriented dialog system with GPT-2. In AAAI, volume 35, pages 14230–14238.
- Adapt-and-distill: Developing small, fast and effective pretrained language models for domains. In Findings of ACL, pages 460–470.
- Differentially private fine-tuning of language models. In ICLR.
- Revisiting few-sample {bert} fine-tuning. In ICLR.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.