2000 character limit reached
Improved Bounds for Pure Private Agnostic Learning: Item-Level and User-Level Privacy
Published 30 Jul 2024 in cs.LG | (2407.20640v2)
Abstract: Machine Learning has made remarkable progress in a wide range of fields. In many scenarios, learning is performed on datasets involving sensitive information, in which privacy protection is essential for learning algorithms. In this work, we study pure private learning in the agnostic model -- a framework reflecting the learning process in practice. We examine the number of users required under item-level (where each user contributes one example) and user-level (where each user contributes multiple examples) privacy and derive several improved upper bounds. For item-level privacy, our algorithm achieves a near optimal bound for general concept classes. We extend this to the user-level setting, rendering a tighter upper bound than the one proved by Ghazi et al. (2023). Lastly, we consider the problem of learning thresholds under user-level privacy and present an algorithm with a nearly tight user complexity.
- User-level differential privacy with few examples per user. In Advances in Neural Information Processing Systems, 2023.
- Calibrating noise to sensitivity in private data analysis. In Proceedings of the 3rd Conference on Theory of Cryptography, pages 265–284, Berlin, Heidelberg, 2006a. Springer.
- Our data, ourselves: Privacy via distributed noise generation. In Proceedings of the 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, pages 486–503, Berlin, Heidelberg, 2006b. Springer.
- Apple Differential Privacy Team. Learning with privacy at scale. Apple Machine Learning Journal, 2017.
- John M Abowd. The U.S. Census Bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 2867. Association for Computing Machinery, 2018.
- What can we learn privately? SIAM Journal on Computing, 40(3):793–826, 2011.
- Leslie G Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, 1984.
- Sample complexity bounds on differentially private learning via communication complexity. In Proceedings of the 27th Conference on Learning Theory, volume 35, pages 1000–1019. PMLR, 2014.
- Private PAC learning implies finite Littlestone dimension. In Proceedings of the 51st Annual ACM Symposium on Theory of Computing, pages 852–860. Association for Computing Machinery, 2019.
- David Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation, 100(1):78–150, 1992.
- Toward efficient agnostic learning. Machine Learning, 17:115–141, 1994.
- Characterizing the sample complexity of pure private learners. Journal of Machine Learning Research, 20:1–33, 2019.
- Learning privately with labeled and unlabeled examples. In Proceedings of the 26th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 461–477. SIAM, 2015.
- Closure properties for private classification and online prediction. In Proceedings of the 33rd Conference on Learning Theory, volume 125, pages 119–152. PMLR, 2020.
- Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
- User-level differentially private learning via correlated sampling. In Advances in Neural Information Processing Systems, volume 34, pages 20172–20184. Curran Associates, Inc., 2021a.
- Hans Ulrich Simon. General bounds on the number of examples needed for learning probabilistic concepts. Journal of Computer and System Sciences, 52(2):239–254, 1996.
- Exact Kolmogorov and total variation distances between some familiar discrete distributions. Journal of Inequalities and Applications, 2006:1–8, 2006.
- Learning discrete distributions: user vs item-level privacy. In Advances in Neural Information Processing Systems, volume 33, pages 20965–20976. Curran Associates, Inc., 2020.
- Bounds on the sample complexity for private learning and private data release. Machine learning, 94:401–437, 2014.
- An equivalence between private classification and online prediction. In Proceedings of the 61st Annual IEEE Symposium on Foundations of Computer Science, pages 389–402, 2020.
- Sample-efficient proper PAC learning with approximate differential privacy. In Proceedings of the 53rd Annual ACM Symposium on Theory of Computing, pages 183–196. Association for Computing Machinery, 2021b.
- Private and online learnability are equivalent. Journal of the ACM, 69(4):1–34, 2022.
- Mechanism design via differential privacy. In Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science, pages 94–103. IEEE Computer Society, 2007.
- Theory of pattern recognition. Nauka, Moscow, 1974.
- Michel Talagrand. Sharper bounds for gaussian and empirical processes. The Annals of Probability, 22:28–76, 1994.
- Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1999.
- VN Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16(2):264–280, 1971.
- Learnability and the Vapnik-Chervonenkis dimension. Journal of the ACM, 36(4):929–965, 1989.
- Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014.
- Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A, 13(1):145–147, 1972.
- Herman Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics, pages 493–507, 1952.
- Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30, 1963.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.