SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning

Abstract: This paper introduces an open-source, decentralized framework named SigmaRL, designed to enhance both sample efficiency and generalization of multi-agent Reinforcement Learning (RL) for motion planning of connected and automated vehicles. Most RL agents exhibit a limited capacity to generalize, often focusing narrowly on specific scenarios, and are usually evaluated in similar or even the same scenarios seen during training. Various methods have been proposed to address these challenges, including experience replay and regularization. However, how observation design in RL affects sample efficiency and generalization remains an under-explored area. We address this gap by proposing five strategies to design information-dense observations, focusing on general features that are applicable to most traffic scenarios. We train our RL agents using these strategies on an intersection and evaluate their generalization through numerical experiments across completely unseen traffic scenarios, including a new intersection, an on-ramp, and a roundabout. Incorporating these information-dense observations reduces training times to under one hour on a single CPU, and the evaluation results reveal that our RL agents can effectively zero-shot generalize. Code: github.com/bassamlab/SigmaRL

• The paper presents a decentralized MARL framework that leverages innovative observation designs to enhance sample efficiency and generalizability.
• It employs a multi-agent extension of PPO with tailored strategies, enabling agents to achieve zero-shot generalization across diverse traffic scenarios.
• Numerical evaluations demonstrate that SigmaRL reduces training time to under an hour while effectively adapting to unseen environments like intersections, on-ramps, and roundabouts.

An Analysis of SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning

The paper "SigmaRL: A Sample-Efficient and Generalizable Multi-Agent Reinforcement Learning Framework for Motion Planning" introduces an open-source, decentralized framework named SigmaRL. This framework is tailored for Connected and Automated Vehicles (CAVs) and is aimed at improving the sample efficiency and generalizability of Multi-Agent Reinforcement Learning (MARL) for motion planning tasks. The study addresses a key limitation within the field of Reinforcement Learning (RL), namely the challenge of generalizing policies to unseen scenarios, in addition to the notorious issue of sample inefficiency.

Summary and Key Contributions

SigmaRL focuses on enhancing RL agents' performance by leveraging an innovative observation design, aiming to bridge the gap between the need for a diverse set of training scenarios and efficient policy learning. The paper identifies a crucial shortcoming in current practices: while existing strategies like experience replay and regularization have been explored to improve generalizability, the design of observations has not received comparable attention, even though it significantly affects sample efficiency.

The contributions of SigmaRL can be summarized as follows:

1. Framework Architecture: SigmaRL employs a multi-agent extension of Proximal Policy Optimization (PPO), designed to operate within a centralized learning and decentralized execution paradigm. This allows agents to share experiences during training while executing their learned policies independently, thus addressing non-stationarity issues in MARL.
2. Observation Design Strategies: The core innovation lies in proposing five observation strategies that involve:
   • Adopting an ego view instead of a bird-eye view for inputs.
   • Observing vertices of surrounding agents in lieu of their poses and geometric dimensions.
   • Observing the distances to surrounding agents and lane boundaries.
   • Observing the distances to lane center lines.

These strategies are designed to offer general features applicable to many traffic scenarios, thereby enhancing agents' sample efficiency and generalization.

1. Numerical Evaluation: The framework's efficacy is demonstrated by training agents on a single traffic scenario and testing them across multiple unseen scenarios, including intersections, on-ramps, and roundabouts, without additional training. The results indicate significant improvements in agents' ability to perform zero-shot generalization, meaning they can successfully navigate new, previously unseen environments based solely on their original training.

Implications and Future Directions

SigmaRL represents a notable step in improving the practicality and adaptability of MARL frameworks in real-world settings, such as CAVs. The introduction of a structured observation framework not only accelerates training (bringing the training time to under an hour) but also permits RL agents to generalize across different scenarios, a critical feature for deployment in dynamic real-world environments.

The study opens up several avenues for future AI developments:

• Scalability and Robustness: Expanding the framework to include more complex environments or higher-dimensional observation spaces could further verify the robustness of the proposed strategies.
• Integration with Real-World Data: Incorporating real-world traffic data into training could enhance the relevance and scalability of SigmaRL beyond simulated environments.
• Comparison with End-to-End Learning: Future studies might explore how SigmaRL's structured observations compare with end-to-end methods utilizing raw sensory data, focusing on interpretability, efficiency, and adaptability.

In conclusion, by proposing a focused yet flexible approach to observation design in RL, this paper contributes to the broader goal of developing autonomous systems capable of learning efficiently and generalizing effectively in realistic, variable conditions.