Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles

Abstract: Role-based learning is a promising approach to improving the performance of Multi-Agent Reinforcement Learning (MARL). Nevertheless, without manual assistance, current role-based methods cannot guarantee stably discovering a set of roles to effectively decompose a complex task, as they assume either a predefined role structure or practical experience for selecting hyperparameters. In this article, we propose a mathematical Structural Information principles-based Role Discovery method, namely SIRD, and then present a SIRD optimizing MARL framework, namely SR-MARL, for multi-agent collaboration. The SIRD transforms role discovery into a hierarchical action space clustering. Specifically, the SIRD consists of structuralization, sparsification, and optimization modules, where an optimal encoding tree is generated to perform abstracting to discover roles. The SIRD is agnostic to specific MARL algorithms and flexibly integrated with various value function factorization approaches. Empirical evaluations on the StarCraft II micromanagement benchmark demonstrate that, compared with state-of-the-art MARL algorithms, the SR-MARL framework improves the average test win rate by 0.17%, 6.08%, and 3.24%, and reduces the deviation by 16.67%, 30.80%, and 66.30%, under easy, hard, and super hard scenarios.

• The paper introduces SR-MARL, which leverages structural information principles to automate hierarchical role discovery in multi-agent reinforcement learning.
• It employs a novel sparsification and optimization process that enhances performance, achieving up to 6.08% win rate improvement and reduced deviation by 66.30%.
• Integration with QMIX and QPLEX demonstrates that SR-MARL improves coordination and adaptability in challenging StarCraft II micromanagement scenarios.

Effective and Stable Role-Based Multi-Agent Collaboration

The paper "Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles" (2304.00755) proposes a novel method for enhancing multi-agent reinforcement learning (MARL) by incorporating structural information principles into role discovery. The proposed framework, SR-MARL, aims to improve performance in cooperative scenarios by addressing common challenges such as scalability and partial observability.

Role Discovery with Structural Information Principles

Framework Overview

The SR-MARL framework is composed of several modules, including a role discovery module (SIRD), which transforms the task of role discovery into hierarchical action space clustering. The framework integrates seamlessly with value function factorization approaches, enabling flexible improvements in multi-agent coordination without manual tuning.

Figure 1: The overall framework of the SR-MARL.

Structural Information-Based Role Discovery (SIRD)

The SIRD method leverages structural information principles to derive an optimal hierarchical clustering of the action space using a multi-stage process:

1. Structuralization: Constructs an action graph where actions are vertices connected by edges representing functional correlations. Action representations are learned using an encoder-decoder structure.
2. Sparsification: Employs one-dimensional structural entropy minimization to reduce the graph's complexity by creating a $k$-nearest neighbor graph.
3. Optimization: Utilizes operators derived from $K$-dimensional structural entropy to iteratively refine the encoding tree, achieving optimal hierarchical clustering for role discovery.

   Figure 2: The structural information principles-based role discovery.

Empirical Evaluation

The SR-MARL framework was evaluated on the StarCraft II micromanagement benchmark, demonstrating performance improvements in various task scenarios (easy, hard, and super hard maps).

Performance Improvements

SR-MARL outperformed existing state-of-the-art MARL algorithms, achieving significant improvements in both average test win rates and stability across different task complexities, especially in high-exploration environments:

• Achieved up to 6.08% improvement in average test win rates.
• Reduced deviation by up to 66.30%, indicating enhanced stability.

  Figure 3: (left) The average test win rates across all 13 maps; (right) the number of maps (out of 13) where the algorithm's average test win rate is the highest.

Integration and Ablation Studies

The framework's integrative capabilities were tested by combining SIRD with existing MARL methods such as QMIX and QPLEX, which resulted in enhanced coordination and learning efficiency.

Ablation studies confirmed the importance of structuralization and sparsification. Variants without these components demonstrated lower performance, emphasizing the necessity of these modules for achieving effective role discovery and performance.

Figure 4: Average test win rates of the SR-MARL integrated with value decomposition methods QMIX and QPLEX.

Conclusion

The paper introduces a robust role discovery method using structural information principles that substantially improves multi-agent collaboration in reinforcement learning. By automating role discovery and optimizing hierarchical action space clustering, SR-MARL delivers a more effective and stable performance. Future research could explore further refinement of the encoding tree and adaptation to diverse MARL environments.