Automatic Intersection Management in Mixed Traffic Using Reinforcement Learning and Graph Neural Networks

Abstract: Connected automated driving has the potential to significantly improve urban traffic efficiency, e.g., by alleviating issues due to occlusion. Cooperative behavior planning can be employed to jointly optimize the motion of multiple vehicles. Most existing approaches to automatic intersection management, however, only consider fully automated traffic. In practice, mixed traffic, i.e., the simultaneous road usage by automated and human-driven vehicles, will be prevalent. The present work proposes to leverage reinforcement learning and a graph-based scene representation for cooperative multi-agent planning. We build upon our previous works that showed the applicability of such machine learning methods to fully automated traffic. The scene representation is extended for mixed traffic and considers uncertainty in the human drivers' intentions. In the simulation-based evaluation, we model measurement uncertainties through noise processes that are tuned using real-world data. The paper evaluates the proposed method against an enhanced first in - first out scheme, our baseline for mixed traffic management. With increasing share of automated vehicles, the learned planner significantly increases the vehicle throughput and reduces the delay due to interaction. Non-automated vehicles benefit virtually alike.

• The paper introduces a novel framework combining RL and GNNs to efficiently manage intersections in mixed traffic environments.
• It employs a centralized POMDP approach and graph-based scene representation to capture complex interactions between automated and manual vehicles.
• Simulation results show significant improvements in vehicle throughput and reduced collision rates compared to traditional eFIFO methods.

Automatic Intersection Management Using RL and GNNs

The paper addresses the challenge of managing intersections in mixed traffic environments, leveraging reinforcement learning (RL) and graph neural networks (GNNs). Traditional intersection management methods fail to account for the complexities introduced by mixed traffic consisting of both automated vehicles (AVs) and manually driven vehicles (MVs). The proposed approach aims to optimize intersection management by considering these varied dynamics, focusing on enhancing traffic flow and reducing interaction-induced delays.

Intersection Management Framework

Reinforcement Learning Paradigm

The problem is framed as a centralized training and execution paradigm to leverage explicit communication between connected vehicles. It is treated as a partially observable Markov decision process (POMDP), facilitating centralized decision-making through a robust communication network. By focusing on the cooperative interactions between AVs and MVs, the RL planner is tuned to negotiate intersection management effectively, ensuring joint optimization of vehicle behavior across different traffic compositions.

Graph-Based Scene Representation

The graph-based input representation uses directed graphs where vertices represent vehicles, and edges encapsulate interaction dynamics, classified by relationship and automation status. The system considers potential collision points and dynamic vehicular priorities influenced by individual road-orientation and destination intentions. Importantly, uncertainties in MV intentions are tactically integrated, allowing the model to adaptively reassess vehicular priorities and provide enriched context to the RL planner.

Network Architecture

The architecture combines relational graph convolutional networks (RGCN) with graph attention layers (GAT) to process the graph-based input, enhancing the model's ability to discern and exploit relevant interactions across different traffic dynamics. By encoding vehicle attributes and conflicts into latent spaces, the model derives joint actions that optimize the interactions. The TD3 algorithm, adapted for continuous action spaces, is deployed to ensure policy robustness and efficiency.

Evaluation in Mixed Traffic Environments

Simulation Setup and Metrics

The training of the RL model is split across varying automation levels, progressively incorporating mixed traffic challenges. Evaluation focuses on metrics such as vehicle throughput, average velocity along differing road preferences, and interaction-induced delays under varied traffic densities and automation levels.

Performance Results

The RL-based intersection management scheme shows marked improvements in vehicle flow rates compared to the enhanced first in - first out (eFIFO) scheme. Particularly in high-demand scenarios, the RL planner significantly enhances throughput, supporting even distributional improvements for AVs and MVs alike. Compared to previous models, the proposed approach maintains optimal performance across different traffic compositions without sacrificing robustness to measurement noise.

Robustness and Limitations

Although the scheme exhibits a substantial reduction in collision rates, ongoing uncertainties modeled through measurement noise processes demonstrate areas for improvement. The model effectively manages decision-making under these uncertainties, presenting a clear advantage over rule-based systems like eFIFO, which show higher susceptibility to collision under similar conditions.

Conclusion

The study introduces an innovative RL and GNN-based framework for automatic intersection management in mixed traffic. By accommodating both AVs and MVs, it addresses real-world demands for intelligent traffic systems. Future efforts will focus on bridging the remaining gaps with real-world deployment, enhancing motion planning algorithms, and practical integration with testing vehicles for real-world scenarios. This advancement promises significant implications for urban mobility, ensuring more efficient and safer intersections.