Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

Abstract: Existing evaluation suites for multi-agent reinforcement learning (MARL) do not assess generalization to novel situations as their primary objective (unlike supervised-learning benchmarks). Our contribution, Melting Pot, is a MARL evaluation suite that fills this gap, and uses reinforcement learning to reduce the human labor required to create novel test scenarios. This works because one agent's behavior constitutes (part of) another agent's environment. To demonstrate scalability, we have created over 80 unique test scenarios covering a broad range of research topics such as social dilemmas, reciprocity, resource sharing, and task partitioning. We apply these test scenarios to standard MARL training algorithms, and demonstrate how Melting Pot reveals weaknesses not apparent from training performance alone.

• The paper presents Melting Pot as a novel evaluation suite that tests zero-shot generalization in multi-agent reinforcement learning using over 80 diverse interaction scenarios.
• The methodology pairs physical environments with pre-trained agents to simulate realistic challenges such as social dilemmas and resource sharing.
• Experimental results reveal that algorithms with collective objectives outperform standard reward maximization methods in complex multi-agent settings.

Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot

The paper introduces Melting Pot, a comprehensive evaluation suite designed for Multi-Agent Reinforcement Learning (MARL). Recognizing the limited scope of existing MARL benchmarks, the authors highlight Melting Pot’s focus on assessing generalization to novel multi-agent interactions. The fundamental proposition is utilizing agents’ interactions as part of the testing environment to innovate on scalability in creating diverse test scenarios.

Key Contributions

1. Novel Evaluation Suite: Melting Pot is framed as an evaluation methodology that moves MARL closer to the rigorous benchmarks prevalent in supervised learning. By emphasizing zero-shot generalization, it reflects real-world multi-agent system requirements, where agents must interact effectively with unknown others in novel settings.
2. Scenario Design: Melting Pot comprises over 80 unique scenarios designed around strategic interactions such as social dilemmas and resource sharing. Each scenario pairs a substrate (physical environment) with a background population (pre-trained agents), excluding focal agents under evaluation to test in these without prior exposure.
3. Multi-Agent Generalization: The authors focus on the diversity and flexibility of Melting Pot scenarios to simulate dynamic, inter-agent dependencies often observed in real-world applications. This presents a practical exploration of multi-agent learning algorithms’ adaptability and robustness.
4. Comprehensive Metrics: Performance in Melting Pot is evaluated not only on task success but also on secondary metrics like impacts on background populations, emphasizing cooperative and fair behavior as reflected by equality and sustainability metrics.

Experimental Results

The paper presents benchmark results using several MARL models, examining the efficiency of algorithms like A3C, V-MPO, and OPRE. It was observed that standard reward maximization strategies may underperform compared to those employing collective objectives, particularly in socially complex scenarios. However, a consistent challenge for current agents is the tendency to overfit to training settings, reducing their effectiveness when faced with novel peer interactions.

Practical and Theoretical Implications

The development of Melting Pot sets a new standard for evaluating MARL by prioritizing generalization. Practically, as MARL systems deploy in varying multi-agent environments, this will offer insights into system robustness, cooperation, and efficiency. Theoretically, Melting Pot could influence future research directions, fostering development in reinforcement learning methods that generalize across a broader range of dynamic multi-agent contexts.

Conclusion and Future Directions

Melting Pot can catalyze advancements in MARL by offering an extensible, scalable platform for rigorous testing. Its open-source nature invites further contributions, potentially expanding to include more complex interactions, such as communication and negotiation. The ongoing evolution of Melting Pot ensures its relevance in developing intelligent multi-agent systems capable of navigating the intricacies of real-world environments.