PAGNet: Pluggable Adaptive Generative Networks for Information Completion in Multi-Agent Communication

Abstract: For partially observable cooperative tasks, multi-agent systems must develop effective communication and understand the interplay among agents in order to achieve cooperative goals. However, existing multi-agent reinforcement learning (MARL) with communication methods lack evaluation metrics for information weights and information-level communication modeling. This causes agents to neglect the aggregation of multiple messages, thereby significantly reducing policy learning efficiency. In this paper, we propose pluggable adaptive generative networks (PAGNet), a novel framework that integrates generative models into MARL to enhance communication and decision-making. PAGNet enables agents to synthesize global states representations from weighted local observations and use these representations alongside learned communication weights for coordinated decision-making. This pluggable approach reduces the computational demands typically associated with the joint training of communication and policy networks. Extensive experimental evaluations across diverse benchmarks and communication scenarios demonstrate the significant performance improvements achieved by PAGNet. Furthermore, we analyze the emergent communication patterns and the quality of generated global states, providing insights into operational mechanisms.

• The paper introduces PAGNet, a framework integrating generative models with multi-agent reinforcement learning communication to help agents synthesize global state representations from weighted local observations.
• PAGNet models communication at an adaptive information level, learning weights for parts of observations using attention to bridge the gap between centralized training and decentralized execution.
• Its pluggable architecture allows decoupling communication module training from sparse RL rewards for efficiency, with experiments showing significant performance gains over state-of-the-art methods.

PAGNet: Pluggable Adaptive Generative Networks for Information Completion in Multi-Agent Communication

PAGNet (Pluggable Adaptive Generative Networks) introduces a novel framework designed to enhance cooperation and communication efficiency in MARL, specifically targeting partially observable cooperative tasks. The architecture uniquely integrates generative models with MARL communication mechanisms within the CTDE paradigm. PAGNet enables agents to synthesize global state representations from weighted local observations and communicated information, addressing partial observability challenges and fostering more coordinated decision-making. The framework emphasizes adaptive, information-level communication modeling and a pluggable design to improve training efficiency and flexibility.

Core Contributions of PAGNet

The paper makes several key contributions to the field of MARL:

1. Integration of Generative Models with MARL Communication: PAGNet innovatively combines generative models and MARL, enabling agents to learn and synthesize global state representations using weighted local observations obtained through communication. These generated states, along with learned communication weights, are subsequently utilized for coordinated decision-making processes.
2. Adaptive Information-Level Communication Modeling: The framework transcends traditional communication protocols by modeling communication at the information level. It learns adaptive weights for specific pieces of information within agents' local observations. This mechanism facilitates effective weighting and integration of communication content tailored to the task at hand, bridging the information gap between centralized training (where global state is often assumed) and decentralized execution (where only local observations are available). This is achieved via a dot-product attention mechanism, where an agent i's local observation (o_t^i) acts as the query, and the collection of all agents' observations (M_t^{i, -i}) serves as keys and values. Positional encoding preserves temporal information, and the network outputs an information-level weight matrix (W_t^i) for each agent i. These weights are then used to create "weighted information" (x_t^i) by selectively masking less important parts of the received information with noise.
3. Pluggable Architecture for Efficiency: PAGNet features a "pluggable" design, where the Information-level Weight Network can be potentially pretrained using generative model loss functions (MSE and GAN loss), decoupling its training from the sparse rewards typical in RL. This enhances the learning efficiency of the policy network and allows seamless integration into various MARL algorithms, supporting transitions between online and offline learning modes.
4. Comprehensive Experimental Validation: The paper provides extensive experimental evaluations across diverse benchmarks (LBF, Hallway, SMAC) and communication scenarios. The results demonstrate significant performance improvements over state-of-the-art MARL methods (both with and without communication) and ablation variants. Visualizations of adaptive weights and generated states offer insights into PAGNet's operational mechanisms.

Mechanisms Underlying PAGNet

PAGNet builds upon value-based MARL within the CTDE framework, incorporating the following key components:

1. Information-level Weight Network (Communication Modeling): The primary function is to model communication content by learning weights for different parts of agents' local observations (o_t^i), effectively treating these observations as messages (m_t^i). This network uses a dot-product attention mechanism.
2. Adaptive Generative Network (Global State Generation): To address partial observability, PAGNet employs a GAN structure. An Information Completion Network (Generator), based on a U-Net architecture, takes the weighted information x_t as input and generates the estimated global state widehat{s}_t. A CNN-based Global Discriminator Network is trained to distinguish between real global states (s_t) and generated states (widehat{s}_t). The network is trained using a combined loss function consisting of a weighted MSE loss and a GAN loss. This training process also updates the Information-level Weight Network, decoupling its learning from sparse RL rewards.
3. Pluggable Module Design: The Information-level Weight Network functions as the pluggable module. Its training, alongside the generative network, relies on GAN/MSE losses, enabling pretraining offline if suitable data is available. During centralized training, the generated global state (widehat{s}_t = G(o_t, W_t)) and the learned weights (W_t) are used by the central mixer network (Q_{tot}) to calculate the TD error. During decentralized execution, agents use the forward pass of the (potentially pretrained) weight network to process communicated information for their individual policy networks (Q^i).
4. Transformer-based Decoder (Policy Network Enhancement): To effectively process the potentially large amount of weighted information (x_t^i) received by each agent after communication, the standard MLP decoder in the individual agent's Q-network (Q^i) is replaced with a Transformer architecture. It uses self-attention to integrate the weighted information and historical context (h_{t-1}^i) to compute the action-values (Q^i) and update the hidden state (h_t^i).

Significance and Experimental Validation

PAGNet significantly advances MARL with communication by directly tackling the challenge of limited local views through the generation of global state representations. This enables more informed and coordinated decisions, moving beyond simple communication protocols to model the content and relevance of information. The pluggable design and decoupling of communication module training from RL rewards enhance overall training efficiency and stability. Furthermore, the framework bridges the gap between centralized training and decentralized execution via consistent communication mechanisms, offering flexibility and applicability through its integration with various MARL algorithms.

Experimental results demonstrate that PAGNet outperforms strong baselines, including QMIX, DGN, G2ANet, and MASIA, across LBF, Hallway, and various SMAC maps, particularly in complex scenarios requiring significant coordination and communication (e.g., Hallway, SMAC hard/super hard maps). Ablation studies confirm the importance of information-level weighting (PAGNet vs. PAGNet_FC) and the benefits of pretraining (PAGNet_PT achieves faster convergence). Visualization of learned information-level weights reveals interpretable and adaptive communication strategies, with agents communicating more intensely during critical moments and reducing communication when less information needs exchanging. Comparisons of generated global states with real states indicate that the adaptive generative network effectively captures essential information about agent positions, health, formations, and enemy presence. Scalability experiments show that PAGNet maintains strong performance even with severely limited visibility, with manageable computational costs comparable to other state-of-the-art communication methods like MASIA.

Conclusion

In conclusion, PAGNet presents a robust approach to integrating generative models into MARL for enhanced communication and cooperation. By modeling communication at the information level, generating global state representations, and employing a pluggable architecture, it effectively addresses key challenges in partially observable multi-agent tasks, resulting in significant performance improvements and enhanced training efficiency.