A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity

Abstract: The key challenge in multiagent learning is learning a best response to the behaviour of other agents, which may be non-stationary: if the other agents adapt their strategy as well, the learning target moves. Disparate streams of research have approached non-stationarity from several angles, which make a variety of implicit assumptions that make it hard to keep an overview of the state of the art and to validate the innovation and significance of new works. This survey presents a coherent overview of work that addresses opponent-induced non-stationarity with tools from game theory, reinforcement learning and multi-armed bandits. Further, we reflect on the principle approaches how algorithms model and cope with this non-stationarity, arriving at a new framework and five categories (in increasing order of sophistication): ignore, forget, respond to target models, learn models, and theory of mind. A wide range of state-of-the-art algorithms is classified into a taxonomy, using these categories and key characteristics of the environment (e.g., observability) and adaptation behaviour of the opponents (e.g., smooth, abrupt). To clarify even further we present illustrative variations of one domain, contrasting the strengths and limitations of each category. Finally, we discuss in which environments the different approaches yield most merit, and point to promising avenues of future research.

• The paper introduces a comprehensive survey of algorithms that address non-stationarity in multiagent systems.
• It presents a novel framework with policy generating functions, belief representation, and influence functions to model adaptive behaviors.
• The study categorizes methods from ignoring to advanced theory of mind, highlighting challenges and future directions for scalable MAS algorithms.

Overview of "A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity"

The paper "A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity" by Hernandez-Leal et al. offers a comprehensive review of the challenges and methods associated with learning in environments where multiple agents interact and adapt. Multiagent systems (MAS) are at the crux of numerous real-world applications, from energy distribution networks to autonomous vehicle coordination. However, the dynamic nature of these systems, where agents continuously adapt and respond to each other's actions, creates a non-stationary environment. This non-stationarity poses a significant challenge for developing efficient learning algorithms.

Problem Definition and Framework

The authors highlight the primary challenge in multiagent learning as the need to continually adapt to non-stationary behaviors induced by other agents. Unlike single-agent environments, the presence of multiple learning agents means that each agent's optimal policy is contingent upon the policies of others, which may change unpredictably.

To address this, the paper presents a novel framework for modeling multiagent environments. This framework encompasses three components:

1. Policy Generating Functions (PGF): These represent the mechanism by which an agent determines its policy based on its history.
2. Belief Representation: This describes an agent's belief about the PGFs of other agents, essentially modeling the probability distribution over their strategies.
3. Influence Function: A mapping from the belief representation to a decision policy, allowing the agent to account for the influence of others' behaviors on its optimal action.

Taxonomic Categorization

Hernandez-Leal et al. categorize algorithms based on how they manage non-stationarity, presenting five increasing levels of sophistication:

• Ignore: Assume environmental stationarity and disregard opponent adaptation.
• Forget: Continuously adapt to changes by de-emphasizing outdated information.
• Respond to Target Opponents: Tailor strategies against specific types of opposing behaviors.
• Learn Opponent Models: Develop models of opponent strategies to better anticipate and adapt to changes.
• Theory of Mind: Assume opponents are reasoning about the agent's strategies and recursively model this reasoning.

Algorithmic Approaches

The paper explores a variety of algorithmic approaches used in addressing non-stationarity within MAS. These include game-theoretic methods focusing on Nash equilibria, reinforcement learning algorithms like WoLF-PHC, and multi-armed bandit frameworks such as Exp3 for adversarial environments. The authors discuss the strengths and limitations of each approach, noting that while some algorithms provide robust theoretical guarantees, they may not be practical in highly dynamic environments.

Implications and Future Directions

The survey underscores the complexity of designing algorithms that can efficiently handle non-stationary behaviors in MAS. It encourages future research to focus on:

• Developing frameworks that effectively balance exploration and exploitation in non-static environments.
• Creating scalable and efficient algorithms capable of handling the intricacies of multiple interacting agents.
• Investigating new domains and application contexts, such as negotiation and smart energy systems.

This paper contributes significantly to the field by not only cataloging existing methods but also proposing a coherent structure for understanding and advancing multiagent learning. As the field progresses, integrating insights from diverse domains such as behavioral game theory, deep learning, and transfer learning may pave the way for more adaptive and resilient MAS solutions.