Papers
Topics
Authors
Recent
Search
2000 character limit reached

Differentially Private Clustered Federated Learning

Published 29 May 2024 in cs.LG, cs.CR, and cs.DC | (2405.19272v6)

Abstract: Federated learning (FL), which is a decentralized ML approach, often incorporates differential privacy (DP) to provide rigorous data privacy guarantees. Previous works attempted to address high structured data heterogeneity in vanilla FL settings through clustering clients (a.k.a clustered FL), but these methods remain sensitive and prone to errors, further exacerbated by the DP noise. This vulnerability makes the previous methods inappropriate for differentially private FL (DPFL) settings with structured data heterogeneity. To address this gap, we propose an algorithm for differentially private clustered FL, which is robust to the DP noise in the system and identifies the underlying clients' clusters correctly. To this end, we propose to cluster clients based on both their model updates and training loss values. Furthermore, for clustering clients' model updates at the end of the first round, our proposed approach addresses the server's uncertainties by employing large batch sizes as well as Gaussian Mixture Models (GMM) to reduce the impact of DP and stochastic noise and avoid potential clustering errors. This idea is efficient especially in privacy-sensitive scenarios with more DP noise. We provide theoretical analysis to justify our approach and evaluate it across diverse data distributions and privacy budgets. Our experimental results show its effectiveness in addressing large structured data heterogeneity in DPFL.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273–1282. PMLR, 2017.
  2. Deep models under the gan: Information leakage from collaborative deep learning. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017.
  3. A survey of privacy attacks in machine learning. ArXiv, 2020.
  4. Beyond inferring class representatives: User-level privacy leakage from federated learning. IEEE INFOCOM, 2019.
  5. Deep leakage from gradients. In Neural Information Processing Systems, 2019.
  6. Inverting gradients - how easy is it to break privacy in federated learning? ArXiv, 2020.
  7. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography. Springer-Verlag, 2006.
  8. Our data, ourselves: Privacy via distributed noise generation. In Proceedings of the 24th Annual International Conference on The Theory and Applications of Cryptographic Techniques, 2006.
  9. Cynthia Dwork. A firm foundation for private data analysis. Commun. ACM, 2011.
  10. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 2014.
  11. Learning differentially private recurrent language models. In ICLR, 2018.
  12. Differentially private federated learning: A client level perspective. ArXiv, 2017.
  13. Local differential privacy-based federated learning for internet of things. IEEE Internet of Things Journal, 8:8836–8853, 2020.
  14. Local privacy and statistical minimax rates. 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1592–1592, 2013.
  15. Minimax optimal procedures for locally private estimation. Journal of the American Statistical Association, 113:182 – 201, 2018.
  16. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Sorelle A. Friedler and Christo Wilson, editors, Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 77–91. PMLR, 23–24 Feb 2018.
  17. Inherent tradeoffs in learning fair representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  18. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, ITCS ’12, page 214–226, New York, NY, USA, 2012. Association for Computing Machinery.
  19. Learning fair representations. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 325–333, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
  20. Equality of opportunity in supervised learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  21. Agnostic federated learning. In International Conference on Machine Learning, pages 4615–4625. PMLR, 2019.
  22. Are all users treated fairly in federated learning systems? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2318–2322, 2021.
  23. Focus: Fairness via agent-awareness for federated learning on heterogeneous data, 2023.
  24. Fair resource allocation in federated learning. In International Conference on Learning Representations, 2020.
  25. Tilted empirical risk minimization. In International Conference on Learning Representations, 2020.
  26. Proportional fairness in federated learning. Transactions on Machine Learning Research, 2023.
  27. Federated multi-task learning. In Neural Information Processing Systems, 2017.
  28. Ditto: Fair and robust federated learning through personalization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6357–6368. PMLR, 18–24 Jul 2021.
  29. Federated multi-task learning under a mixture of distributions. In Neural Information Processing Systems, 2021.
  30. Personalized federated learning under mixture of distributions. ArXiv, abs/2305.01068, 2023.
  31. Fedmd: Heterogenous federated learning via model distillation. ArXiv, abs/1910.03581, 2019.
  32. A secure federated transfer learning framework. IEEE Intelligent Systems, 35:70–82, 2020.
  33. Three approaches for personalization with applications to federated learning, 2020.
  34. An efficient framework for clustered federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19586–19597. Curran Associates, Inc., 2020.
  35. Fedsoft: Soft clustered federated learning with proximal local updating. CoRR, abs/2112.06053, 2021.
  36. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 32:3710–3722, 2019.
  37. Provably personalized and robust federated learning, 2023.
  38. Federated learning with hierarchical clustering of local updates to improve training on non-iid data, 2020.
  39. Multi-center federated learning. CoRR, abs/2005.01026, 2020.
  40. Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy. Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, 2020.
  41. Differential privacy and fairness in decisions and learning tasks: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, jul 2022.
  42. Differential privacy has disparate impact on model accuracy. In Neural Information Processing Systems, 2019.
  43. Removing disparate impact on model accuracy in differentially private stochastic gradient descent. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
  44. Disparate impact in differential privacy from gradient misalignment. ArXiv, abs/2206.07737, 2022.
  45. Differentially private and fair deep learning: A lagrangian dual approach. ArXiv, abs/2009.12562, 2020.
  46. Privfairfl: Privacy-preserving group fairness in federated learning, 2022.
  47. Enforcing fairness in private federated learning via the modified method of differential multipliers. ArXiv, abs/2109.08604, 2021.
  48. An axiomatic theory of fairness in network resource allocation. In 2010 Proceedings IEEE INFOCOM, pages 1–9, mar 2010. ISSN: 0743-166X.
  49. The price of fairness. Operations research, 59(1):17–31, 2011.
  50. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016.
  51. Asymptotic convergence rate of the em algorithm for gaussian mixtures. Neural Computation, 2000.
  52. Augment your batch: better training with larger batches, 2019.
  53. Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 2012.
  54. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, 2017.
  55. Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
  56. Advances and open problems in federated learning. Foundations and trends® in machine learning, 14(1–2):1–210, 2021.
  57. Differentially private federated learning on heterogeneous data. In International Conference on Artificial Intelligence and Statistics, 2021.
  58. Differentially private empirical risk minimization under the fairness lens. In Neural Information Processing Systems, 2021.
  59. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14):7684–7689, 2020.
  60. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 0 likes about this paper.

Sign Up for Free