Papers
Topics
Authors
Recent
Search
2000 character limit reached

Networked Communication for Decentralised Agents in Mean-Field Games

Published 5 Jun 2023 in cs.MA, cs.AI, cs.LG, cs.SI, cs.SY, and eess.SY | (2306.02766v5)

Abstract: We introduce networked communication to the mean-field game framework, in particular to oracle-free settings where $N$ decentralised agents learn along a single, non-episodic run of the empirical system. We prove that our architecture has sample guarantees bounded between those of the centralised- and independent-learning cases. We provide the order of the difference in these bounds in terms of network structure and number of communication rounds, and also contribute a policy-update stability guarantee. We discuss how the sample guarantees of the three theoretical algorithms do not actually result in practical convergence. We therefore show that in practical settings where the theoretical parameters are not observed (leading to poor estimation of the Q-function), our communication scheme considerably accelerates learning over the independent case, often performing similarly to a centralised learner while removing the restrictive assumption of the latter. We contribute further practical enhancements to all three theoretical algorithms, allowing us to present their first empirical demonstrations. Our experiments confirm that we can remove several of the theoretical assumptions of the algorithms, and display the empirical convergence benefits brought by our new networked communication. We additionally show that our networked approach has significant advantages over both alternatives in terms of robustness to update failures and to changes in population size.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (119)
  1. Mean Field Games: Numerical Methods. SIAM Journal on Numerical Analysis, 48(3):1136–1162, 2010.
  2. Mean Field Games and Applications: Numerical Aspects, 2020.
  3. Regularization of the policy updates for stabilizing Mean Field Games, 2023.
  4. Q-learning in regularized mean-field games. Dynamic Games and Applications, 13(1):89–117, 2023.
  5. Unified Reinforcement Q-Learning for Mean Field Game and Control Problems, 2021.
  6. Crowd-Averse Cyber-Physical Systems: The Paradigm of Robust Mean-Field Games. IEEE Transactions on Automatic Control, 61(8):2312–2317, 2016.
  7. Robust Mean Field Games with Application to Production of an Exhaustible Resource. IFAC Proceedings Volumes, 45(13):454–459, 2012. 7th IFAC Symposium on Robust Control Design.
  8. Robust mean field games. Dynamic games and applications, 6(3):277–303, 2016.
  9. Dota 2 with Large Scale Deep Reinforcement Learning, 2019.
  10. Adaptive Dynamical Networks, 2023.
  11. Generalized conditional gradient and learning in potential mean field games, 2021.
  12. Proximal methods for stationary Mean Field Games with local couplings. SIAM Journal on Control and Optimization, 56:801–, 03 2018.
  13. A Comprehensive Survey of Multiagent Reinforcement Learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):156–172, 2008.
  14. A policy iteration method for mean field games. ESAIM: COCV, 27:85, 2021.
  15. Embodied evolution of self-organised aggregation by cultural propagation. In Marco Dorigo, Mauro Birattari, Christian Blum, Anders L. Christensen, Andreagiovanni Reina, and Vito Trianni, editors, Swarm Intelligence, pages 351–359, Cham, 2018. Springer International Publishing.
  16. Language Evolution in Swarm Robotics: A Perspective. Frontiers in Robotics and AI, 7, 2020.
  17. Cultural evolution of probabilistic aggregation in synthetic swarms. Applied Soft Computing, 113:108010, 2021.
  18. Emergent Communication through Negotiation. In International Conference on Learning Representations, 2018.
  19. Connecting GANs, MFGs, and OT, 2021.
  20. Learning in mean field games: The fictitious play. ESAIM: COCV, 23(2):569–591, 2017.
  21. E. Carlini and F. J. Silva. A Fully Discrete Semi-Lagrangian Scheme for a First Order Mean Field Game Problem. SIAM Journal on Numerical Analysis, 52(1):45–67, 2014.
  22. Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance, 2021.
  23. Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning, 2021.
  24. Hierarchical multi-robot navigation and formation in unknown environments via deep reinforcement learning and distributed optimization. Robotics and Computer-Integrated Manufacturing, 83:102570, 2023.
  25. Distributed Learning in Wireless Networks: Recent Progress and Future Challenges, 2021.
  26. Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning, 2021.
  27. A Survey on Large-Population Systems and Scalable Multi-Agent Reinforcement Learning. arXiv preprint arXiv:2209.03859, 2022.
  28. Multi-Agent Reinforcement Learning via Mean Field Control: Common Noise, Major Agents and Approximation Properties, 2023.
  29. The Complexity of Computing a Nash Equilibrium. In Proceedings of the Thirty-Eighth Annual ACM Symposium on Theory of Computing, STOC ’06, page 71–78, New York, NY, USA, 2006. Association for Computing Machinery.
  30. Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning, 2019.
  31. What Is an Evolutionary Algorithm?, pages 25–48. Springer Berlin Heidelberg, Berlin, Heidelberg, 2015.
  32. On the Convergence of Model Free Learning in Mean Field Games. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7143–7150, Apr. 2020.
  33. Revisiting Fundamentals of Experience Replay. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
  34. Influence of Local Selection and Robot Swarm Density on the Distributed Evolution of GRNs. In Paul Kaufmann and Pedro A. Castillo, editors, Applications of Evolutionary Computation, pages 567–582, Cham, 2019. Springer International Publishing.
  35. Maintaining Diversity in Robot Swarms with Distributed Embodied Evolution. In Marco Dorigo, Mauro Birattari, Christian Blum, Anders L. Christensen, Andreagiovanni Reina, and Vito Trianni, editors, Swarm Intelligence, pages 395–402, Cham, 2018. Springer International Publishing.
  36. Deep Learning Methods for Mean Field Control Problems With Delay. Frontiers in Applied Mathematics and Statistics, 6, 2020.
  37. Concave Utility Reinforcement Learning: the Mean-Field Game Viewpoint, 2022.
  38. Numerical resolution of McKean-Vlasov FBSDEs using neural networks, 2022.
  39. Generic Behaviour Similarity Measures for Evolutionary Swarm Robotics. In Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, GECCO ’13, page 199–206, New York, NY, USA, 2013. Association for Computing Machinery.
  40. Learning Mean-Field Games. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  41. Learning Mean-Field Games, 2019.
  42. A General Framework for Learning Mean-Field Games, 2020.
  43. Entropy Regularization for Mean Field Games with Learning. Math. Oper. Res., 47(4):3239–3260, nov 2022.
  44. Combining environment-driven adaptation and task-driven optimisation in evolutionary robotics. PloS one, 9(6):e98466, 2014.
  45. Improving Survivability in Environment-Driven Distributed Evolutionary Algorithms through Explicit Relative Fitness and Fitness Proportionate Communication. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO ’15, page 169–176, New York, NY, USA, 2015. Association for Computing Machinery.
  46. Finite-Sample Analysis of Distributed Q-learning for Multi-Agent Networks. In 2020 American Control Conference (ACC), pages 3511–3516, 2020.
  47. Dynamic traffic signal control using mean field multi-agent reinforcement learning in large scale road-networks. IET Intelligent Transport Systems, 04 2023.
  48. Robust Mean Field Linear-Quadratic-Gaussian Games with Unknown L2superscript𝐿2L^{2}italic_L start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT-Disturbance. SIAM Journal on Control and Optimization, 55(5):2811–2840, 2017.
  49. Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle. Communications in Information & Systems, 6(3):221 – 252, 2006.
  50. A game-theoretic framework for autonomous vehicles velocity control: Bridging microscopic differential games and macroscopic mean field games. Discrete and Continuous Dynamical Systems - B, 25(12):4869–4903, 2020.
  51. Coordination of groups of mobile autonomous agents using nearest neighbor rules. IEEE Transactions on Automatic Control, 48(6):988–1001, 2003.
  52. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning, 2019.
  53. 𝒬⁢𝒟𝒬𝒟{{\cal Q}{\cal D}}caligraphic_Q caligraphic_D-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus+InnovationsConsensusInnovations{\rm Consensus}+{\rm Innovations}roman_Consensus + roman_Innovations. IEEE Transactions on Signal Processing, 61(7):1848–1862, 2013.
  54. How Well Do Reinforcement Learning Approaches Cope With Disruptions? The Case of Traffic Signal Control. IEEE Access, 11:36504–36515, 2023.
  55. Simple and Optimal Methods for Stochastic Variational Inequalities, II: Markovian Noise and Policy Evaluation in Reinforcement Learning. SIAM Journal on Optimization, 32(2):1120–1155, 2022.
  56. Mean Field Games. Japanese Journal of Mathematics, 2(1):229–260, 2007.
  57. Learning Mean Field Games: A Survey, 2022.
  58. Scalable Deep Reinforcement Learning Algorithms for Mean Field Games, 2022.
  59. Mathieu Laurière. Numerical Methods for Mean Field Games and Mean Field Type Control, 2021.
  60. Multi-Agent Reinforcement Learning in Sequential Social Dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’17, page 464–473, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
  61. Decentralized Reinforcement Learning of Robot Behaviors. Artificial Intelligence, 256:130–159, 2018.
  62. A Mean Field Games Model for Cryptocurrency Mining, 2022.
  63. A Communication-Efficient Multi-Agent Actor-Critic Algorithm for Distributed Reinforcement Learning. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 5562–5567, 2019.
  64. Long-Ji Lin. Self-Improving Reactive Agents Based on Reinforcement Learning, Planning and Teaching. Mach. Learn., 8(3–4):293–321, may 1992.
  65. Multi-Agent Reinforcement Learning based UAV Swarm Communications Against Jamming. IEEE Transactions on Wireless Communications, pages 1–1, 2023.
  66. An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control, pages 47–66. Springer International Publishing, Cham, 2016.
  67. A mean-field game approach to cloud resource management with function approximation. In Proceedings of the 36th Conference on Advances in Neural Information Processing Systems (NIPS 2022), volume 36, pages 1–12, New Orleans, LA, USA, 2022. Curran Associates, Inc.
  68. Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20238–20248. Curran Associates, Inc., 2020.
  69. Social diversity and social preferences in mixed-motive reinforcement learning, 2020.
  70. Optimal dynamic information provision in traffic routing. CoRR, abs/2001.03232, 2020.
  71. Decentralised Learning in Systems With Many, Many Strategic Agents. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
  72. Model-free Reinforcement Learning for Non-stationary Mean Field Games. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 1032–1037, 2020.
  73. Model-free Reinforcement Learning for Mean Field Games. IEEE Transactions on Control of Network Systems, pages 1–11, 2023.
  74. Linear Quadratic Risk-Sensitive and Robust Mean Field Games. IEEE Transactions on Automatic Control, 62(3):1062–1077, 2017.
  75. Mean Field Behavior of Collaborative Multiagent Foragers. IEEE Transactions on Robotics, 38(4):2151–2165, 2022.
  76. Multi-Agent Deep Reinforcement Learning for Multi-Robot Applications: A Survey. Sensors, 23(7), 2023.
  77. Scaling Mean Field Games by Online Mirror Descent. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’22, page 1028–1037, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems.
  78. Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc.
  79. Mean field games flock! the reinforcement learning way. In IJCAI, 2021.
  80. Generalization in mean field games by learning master policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 9413–9421, 2022.
  81. Real-time optimization of dynamic problems through distributed embodied evolution. Integrated Computer-Aided Engineering, 23(3):237–253, 2016.
  82. Scaling up Mean Field Games with Online Mirror Descent, 2021.
  83. Distributed Averaging in Dynamic Networks. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’10, page 369–370, New York, NY, USA, 2010. Association for Computing Machinery.
  84. Markov game approach for multi-agent competitive bidding strategies in electricity market. IET Generation, Transmission & Distribution, 10:3756–3763(7), November 2016.
  85. Markov–Nash Equilibria in Mean-Field Games with Discounted Cost. SIAM Journal on Control and Optimization, 56(6):4256–4287, 2018.
  86. The StarCraft Multi-Agent Challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, page 2186–2188, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems.
  87. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016.
  88. A multi-agent deep reinforcement learning framework for algorithmic trading in financial markets. Expert Systems with Applications, 208:118124, 2022.
  89. Massive Autonomous UAV Path Planning: A Neural Network Based Mean-Field Game Theoretic Approach, 2019.
  90. Divergence-Regularized Multi-Agent Actor-Critic. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 20580–20603. PMLR, 17–23 Jul 2022.
  91. Reinforcement Learning in Stationary Mean-Field Games. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, page 251–259, Richland, SC, 2019. International Foundation for Autonomous Agents and Multiagent Systems.
  92. Partially Observable Mean Field Reinforcement Learning, 2020.
  93. Decentralized Mean Field Games, 2021.
  94. Multi Type Mean Field Reinforcement Learning, 2022.
  95. A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning, 2019.
  96. Reinforcement Learning: An Introduction. MIT press, 2018.
  97. Mean-Field Learning: a Survey, 2012.
  98. A Robust Mean-field Game of Boltzmann-Vlasov-like Traffic Flow. In 2022 American Control Conference (ACC), pages 556–561, 2022.
  99. Mean field limit of a behavioral financial market model. Physica A: Statistical Mechanics and its Applications, 505:613–631, 2018.
  100. Embodied Evolution for Collective Indoor Surveillance and Location. In Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO Companion ’15, page 1241–1242, New York, NY, USA, 2015. Association for Computing Machinery.
  101. Oracle-free Reinforcement Learning in Mean-Field Games along a Single Sample Path, 2023.
  102. A functional mirror ascent view of policy gradient methods with function approximation. CoRR, abs/2108.05828, 2021.
  103. Munchausen Reinforcement Learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 4235–4246. Curran Associates, Inc., 2020.
  104. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/, 2019.
  105. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, pages 1–5, 2019.
  106. Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 9672–9683, Red Hook, NY, USA, 2018. Curran Associates Inc.
  107. Breaking the Curse of Many Agents: Provable Mean Embedding Q-Iteration for Mean-Field Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
  108. Learning While Playing in Mean-Field Games: Convergence and Optimality. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11436–11447. PMLR, 18–24 Jul 2021.
  109. Distributed Interference-Aware Power Control in Ultra-Dense Small Cell Networks: A Robust Mean Field Game. IEEE Access, 6:12608–12619, 2018.
  110. Mean Field Multi-Agent Reinforcement Learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5571–5580. PMLR, 10–15 Jul 2018.
  111. Policy Mirror Ascent for Efficient and Independent Learning in Mean Field Games. In International Conference on Machine Learning, pages 39722–39754. PMLR, 2023.
  112. Independent Learning and Subjectivity in Mean-Field Games. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 2845–2850, 2022.
  113. Independent Learning in Mean-Field Games: Satisficing Paths and Convergence to Subjective Equilibria, 2022.
  114. A deeper look at experience replay. arXiv preprint arXiv:1712.01275, 2017.
  115. Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5872–5881. PMLR, 10–15 Jul 2018.
  116. Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advances, 2019.
  117. Distributed learning of average belief over networks using sequential observations. Automatica, 115:108857, 2020.
  118. “Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms", pages 321–384. Springer International Publishing, Cham, 2021.
  119. MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free