Papers
Topics
Authors
Recent
Search
2000 character limit reached

FedPower: Privacy-Preserving Distributed Eigenspace Estimation

Published 1 Mar 2021 in stat.ML and cs.LG | (2103.00704v2)

Abstract: Eigenspace estimation is fundamental in machine learning and statistics, which has found applications in PCA, dimension reduction, and clustering, among others. The modern machine learning community usually assumes that data come from and belong to different organizations. The low communication power and the possible privacy breaches of data make the computation of eigenspace challenging. To address these challenges, we propose a class of algorithms called \textsf{FedPower} within the federated learning (FL) framework. \textsf{FedPower} leverages the well-known power method by alternating multiple local power iterations and a global aggregation step, thus improving communication efficiency. In the aggregation, we propose to weight each local eigenvector matrix with {\it Orthogonal Procrustes Transformation} (OPT) for better alignment. To ensure strong privacy protection, we add Gaussian noise in each iteration by adopting the notion of \emph{differential privacy} (DP). We provide convergence bounds for \textsf{FedPower} that are composed of different interpretable terms corresponding to the effects of Gaussian noise, parallelization, and random sampling of local machines. Additionally, we conduct experiments to demonstrate the effectiveness of our proposed algorithms.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (92)
  1. Lazysvd: even faster svd decomposition yet without agonizing pain. In Advances in Neural Information Processing Systems, pages 974–982, 2016.
  2. Differentially private covariance estimation. Advances in Neural Information Processing Systems, 32, 2019.
  3. Federated collaborative filtering for privacy-preserving personalized recommendation system. arXiv preprint arXiv:1901.09888, 2019.
  4. Peter Arbenz. Lecture notes on solving large scale eigenvalue problems. 2012.
  5. Stochastic optimization of pca with capped msg. In Advances in Neural Information Processing Systems, pages 1815–1823, 2013.
  6. Principal components estimation and identification of static factors. Journal of Econometrics, 176(1):18–29, 2013.
  7. An improved gap-dependency analysis of the noisy power method. In Conference on Learning Theory, pages 284–309, 2016.
  8. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in Neural Information Processing Systems, 31, 2018.
  9. On distributed averaging for stochastic k-PCA. In Advances in Neural Information Processing Systems, pages 11024–11033, 2019.
  10. Protection against reconstruction and its applications in private federated learning. arXiv preprint arXiv:1812.00984, 2018.
  11. Composable and versatile privacy via truncated cdp. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, pages 74–86, 2018.
  12. Optimal estimation and rank detection for sparse spiked covariance matrices. Probability theory and related fields, 161(3-4):781–815, 2015.
  13. Exact matrix completion via convex optimization. Foundations of Computational mathematics, 9(6):717, 2009.
  14. Joshua Cape. Orthogonal procrustes and norm-dependent optimality. The Electronic Journal of Linear Algebra, 36(36):158–168, 2020.
  15. Secure federated matrix factorization. IEEE Intelligent Systems, 36(5):11–20, 2020.
  16. Practical lossless federated singular vector decomposition over billion-scale data. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 46–55, 2022.
  17. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069, 2018.
  18. Communication-efficient distributed eigenspace estimation. SIAM Journal on Mathematics of Data Science, 3(4):1067–1092, 2021.
  19. Near-optimal differentially private principal components. In Advances in Neural Information Processing Systems, pages 989–997, 2012.
  20. Distributed estimation for principal component analysis: An enlarged eigenspace analysis. Journal of the American Statistical Association, pages 1–12, 2021.
  21. Accelerated stochastic power iteration. Proceedings of machine learning research, 84:58, 2018.
  22. Gaussian differential privacy. arXiv preprint arXiv:1905.02383, 2019.
  23. Differentially private covariance revisited. arXiv preprint arXiv:2205.14324, 2022.
  24. Concentrated differential privacy. arXiv preprint arXiv:1603.01887, 2016.
  25. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265–284. Springer, 2006.
  26. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211–407, 2014a.
  27. Analyze gauss: optimal bounds for privacy-preserving principal component analysis. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pages 11–20, 2014b.
  28. Exposed! a survey of attacks on private data. Annal Review of Statistics and its Applications, 2017.
  29. Farmtest: Factor-adjusted robust multiple testing with approximate false discovery control. Journal of the American Statistical Association, 114(528):1880–1893, 2019a.
  30. Distributed estimation of principal eigenspaces. The Annals of Statistics, 47(6):3009–3031, 2019b.
  31. Fast and communication-efficient distributed PCA. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7450–7454. IEEE, 2019.
  32. Fast and simple PCA via convex optimization. arXiv preprint arXiv:1509.05647, 2015.
  33. Faster eigenvector computation via shift-and-invert preconditioning. In ICML, pages 2626–2634, 2016.
  34. Communication-efficient algorithms for distributed stochastic principal component analysis. arXiv preprint arXiv:1702.08169, 2017.
  35. Minimax-optimal privacy-preserving sparse PCA in distributed systems. In International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 2018.
  36. Revisiting the nyström method for improved large-scale machine learning. The Journal of Machine Learning Research, 17(1):3977–4041, 2016.
  37. Oded Goldreich. The Foundations of Cryptography, volume 2. Cambridge University Press, 2009.
  38. Calculating the singular values and pseudo-inverse of a matrix. Journal of the Society for Industrial and Applied Mathematics, Series B: Numerical Analysis, 2(2):205–224, 1965.
  39. Gene H Golub and C Reinsch. Singular value decomposition and least squares solutions. Numerische Mathematik, 14:403–420, 1970.
  40. Matrix computations, volume 3. JHU Press, 2012.
  41. Federated principal component analysis. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020), 2020.
  42. Randomized spectral co-clustering for large-scale directed networks. arXiv preprint arXiv:2004.12164, 2020.
  43. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217–288, 2011.
  44. The noisy power method: A meta algorithm with applications. In Advances in Neural Information Processing Systems, pages 2861–2869, 2014.
  45. Beyond worst-case analysis in private singular vector computation. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing, pages 331–340, 2013.
  46. Alan Julian Izenman. Modern multivariate statistical techniques. Regression, classification and manifold learning, 10:978–0, 2008.
  47. Sun Ji-Guang. Perturbation of angles between linear subspaces. Journal of Computational Mathematics, pages 58–61, 1987.
  48. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  49. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020.
  50. First analysis of local GD on heterogeneous data. arXiv preprint arXiv:1909.04715, 2019.
  51. Bias-adjusted spectral clustering in multi-layer stochastic block models. Journal of the American Statistical Association, pages 1–13, 2022.
  52. Differentially private meta-learning. arXiv preprint arXiv:1909.05830, 2019a.
  53. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429–450, 2020a.
  54. Delayed projection techniques for linearly constrained problems: Convergence rates, acceleration, and applications. arXiv preprint arXiv:2101.01505, 2021.
  55. Communication efficient decentralized training with multiple local updates. arXiv preprint arXiv:1910.09126, 2019b.
  56. On the convergence of FedAvg on non-iid data. In International Conference on Learning Representations, 2020b.
  57. Communication-efficient distributed svd via local power iterations. In International Conference on Machine Learning, pages 6504–6514. PMLR, 2021.
  58. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017.
  59. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017.
  60. Learning differentially private recurrent language models. 2018.
  61. Inference attacks against collaborative learning. arXiv preprint arXiv:1805.04049, 13, 2018.
  62. Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263–275. IEEE, 2017.
  63. Randomized block Krylov methods for stronger and faster approximate singular value decomposition. In Advances in Neural Information Processing Systems (NIPS), 2015.
  64. Rank centrality: Ranking from pairwise comparisons. Operations Research, 65(1):266–287, 2017.
  65. On stochastic approximation of the eigenvectors and eigenvalues of the expectation of a random matrix. Journal of mathematical analysis and applications, 106(1):69–84, 1985.
  66. Yousef Saad. Numerical methods for large eigenvalue problems. preparation. Available from: http://www-users. cs. umn. edu/saad/books. html, 2011.
  67. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 31(9):3400–3413, 2019.
  68. Peter H Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10, 1966.
  69. Ohad Shamir. A stochastic pca and svd algorithm with an exponential convergence rate. In International Conference on Machine Learning, pages 144–152, 2015.
  70. Ohad Shamir. Convergence of stochastic gradient descent for pca. In International Conference on Machine Learning, pages 257–265, 2016.
  71. Privately learning subspaces. Advances in Neural Information Processing Systems, 34:1312–1324, 2021.
  72. Federated multi-task learning. In Advances in Neural Information Processing Systems, pages 4424–4434, 2017.
  73. Sebastian U Stich. Local SGD converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018.
  74. Ji-Guang Sun. On perturbation bounds for the QR factorization. Linear algebra and its applications, 215:95–111, 1995.
  75. Joel A Tropp. An introduction to matrix concentration inequalities. arXiv preprint arXiv:1501.01571, 2015.
  76. Jalaj Upadhyay. The price of privacy for low-rank factorization. Advances in Neural Information Processing Systems, 31, 2018.
  77. Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17(4):395–416, 2007.
  78. Minimax sparse principal subspace estimation in high dimensions. The Annals of Statistics, 41(6):2905–2947, 2013.
  79. Cooperative SGD: A unified framework for the design and analysis of communication-efficient SGD algorithms. arXiv preprint arXiv:1808.07576, 2018.
  80. SPSD matrix approximation vis column selection: Theories, algorithms, and extensions. The Journal of Machine Learning Research, 17(1):1697–1745, 2016.
  81. Scalable kernel k-means clustering with nyström approximation: relative-error bounds. The Journal of Machine Learning Research, 20(1):431–479, 2019.
  82. Andreas Winkelbauer. Moments and absolute moments of the normal distribution. arXiv preprint arXiv:1209.4340, 2012.
  83. Randomized algorithms for low-rank matrix factorizations: sharp performance bounds. Algorithmica, 72(1):264–281, 2015.
  84. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37–52, 1987.
  85. David P Woodruff. Sketching as a tool for numerical linear algebra. arXiv preprint arXiv:1411.4357, 2014.
  86. A review of distributed algorithms for principal component analysis. Proceedings of the IEEE, 106(8):1321–1340, 2018.
  87. Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different dimensions. SIAM Journal on Matrix Analysis and Applications, 37(3):1176–1197, 2016.
  88. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In AAAI Conference on Artificial Intelligence, 2019.
  89. Randomized spectral clustering in large-scale stochastic block models. Journal of Computational and Graphical Statistics, 31(3):887–906, 2022.
  90. Federated f-differential privacy. In International Conference on Artificial Intelligence and Statistics, pages 2251–2259. PMLR, 2021.
  91. On the convergence properties of a k-step averaging stochastic gradient descent algorithm for nonconvex optimization. arXiv preprint arXiv:1708.01012, 2017.
  92. Differentially private distributed learning. INFORMS Journal on Computing, 32(3):779–789, 2020.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up