Scalable Decentralized Algorithms for Online Personalized Mean Estimation
Abstract: In numerous settings, agents lack sufficient data to directly learn a model. Collaborating with other agents may help, but it introduces a bias-variance trade-off, when local data distributions differ. A key challenge is for each agent to identify clients with similar distributions while learning the model, a problem that remains largely unresolved. This study focuses on a simplified version of the overarching problem, where each agent collects samples from a real-valued distribution over time to estimate its mean. Existing algorithms face impractical space and time complexities (quadratic in the number of agents A). To address scalability challenges, we propose a framework where agents self-organize into a graph, allowing each agent to communicate with only a selected number of peers r. We introduce two collaborative mean estimation algorithms: one draws inspiration from belief propagation, while the other employs a consensus-based approach, with complexity of O( r |A| log |A|) and O(r |A|), respectively. We establish conditions under which both algorithms yield asymptotically optimal estimates and offer a theoretical characterization of their performance.
- Machine learning and data analytics for the iot. Neural Computing and Applications, 32(20):16205–16233, May 2020. ISSN 1433-3058. doi: 10.1007/s00521-020-04874-y.
- Amini, H. Bootstrap percolation and diffusion in random graphs with given vertex degrees. The Electronic Journal of Combinatorics, 17(1), February 2010. doi: 10.37236/297.
- Collaborative Algorithms for Online Personalized Mean Estimation. 2022. ISSN 2835-8856.
- Waffle: Weighted averaging for personalized federated learning, 2021.
- Linear speedup in personalized collaborative learning, 2022.
- Deng, L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141–142, 2012.
- Digital Library of Mathematical Functions. Asymptotic expansions: Exponential and logarithmic integral, 2023. [Online; accessed 23-January-2024].
- Collaborative Learning by Detecting Collaboration Partners. 35:15629–15641, 2022.
- Model-sharing Games: Analyzing Federated Learning Under Voluntary Participation. volume 35, pp. 5303–5311, 2021.
- On Sample Optimality in Personalized Collaborative and Federated Learning. volume 35, pp. 212–225, 2022.
- Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3557–3568. Curran Associates, Inc., 2020.
- An Efficient Framework for Clustered Federated Learning. In Advances in Neural Information Processing Systems, volume 33, pp. 19586–19597. Curran Associates, Inc., 2020.
- Optimal model averaging: Towards personalized collaborative learning, 2021.
- Janson, S. The probability that a random multigraph is simple. Combinatorics, Probability and Computing, 18(1-2):205–225, 2009. doi: 10.1017/S0963548308009644.
- Advances and Open Problems in Federated Learning. 14(1–2):1–210, 2021. ISSN 1935-8237, 1935-8245. doi: 10.1561/2200000083.
- Federated Learning: Challenges, Methods, and Future Directions, 2020. ISSN 1053-5888, 1558-0792. URL http://arxiv.org/abs/1908.07873.
- Ditto: Fair and robust federated learning through personalization. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 6357–6368. PMLR, 18–24 Jul 2021.
- Maillard, O.-A. Mathematics of Statistical Sequential Decision Making. Habilitation à diriger des recherches, Université de Lille Nord de France, February 2019.
- Three Approaches for Personalization with Applications to Federated Learning, 2020.
- Federated multi-task learning under a mixture of distributions. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
- Uniform generation of random regular graphs of moderate degree. Journal of Algorithms, 11(1):52–67, 1990. ISSN 0196-6774. doi: https://doi.org/10.1016/0196-6774(90)90029-E.
- Meyer, C. D. Matrix Analysis and Applied Linear Algebra. SIAM, 2001.
- Threshold models of cascades in large-scale networks. IEEE Transactions on Network Science and Engineering, 6(2):158–172, 2019. doi: 10.1109/TNSE.2017.2777941.
- Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 32(8):3710–3722, 2021. doi: 10.1109/TNNLS.2020.3015958.
- Understanding Machine Learning - From Theory to Algorithms. Cambridge University Press, 2014. ISBN 978-1-10-705713-5.
- Towards Personalized Federated Learning. 34(12):9587–9603, 2023. ISSN 2162-237X, 2162-2388. doi: 10.1109/TNNLS.2022.3160699.
- Vershynin, R. High-dimensional probability: An introduction with applications in data science. Cambridge University Press, 2018.
- Fast linear iterations for distributed averaging. Systems & Control Letters, 53(1):65–78, 2004. ISSN 0167-6911. doi: https://doi.org/10.1016/j.sysconle.2004.02.022.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.