Power Grid Congestion Management via Topology Optimization with AlphaZero

Abstract: The energy sector is facing rapid changes in the transition towards clean renewable sources. However, the growing share of volatile, fluctuating renewable generation such as wind or solar energy has already led to an increase in power grid congestion and network security concerns. Grid operators mitigate these by modifying either generation or demand (redispatching, curtailment, flexible loads). Unfortunately, redispatching of fossil generators leads to excessive grid operation costs and higher emissions, which is in direct opposition to the decarbonization of the energy sector. In this paper, we propose an AlphaZero-based grid topology optimization agent as a non-costly, carbon-free congestion management alternative. Our experimental evaluation confirms the potential of topology optimization for power grid operation, achieves a reduction of the average amount of required redispatching by 60%, and shows the interoperability with traditional congestion management methods. Our approach also ranked 1st in the WCCI 2022 Learning to Run a Power Network (L2RPN) competition. Based on our findings, we identify and discuss open research problems as well as technical challenges for a productive system on a real power grid.

• The paper demonstrates that integrating AlphaZero with grid topology optimization can significantly reduce redispatch interventions and operational costs.
• It introduces a novel reinforcement learning framework employing a tailored MCTS approach to enhance grid stability in simulation environments.
• The study highlights potential for integrating topology optimization with human-in-the-loop systems to support decarbonization and improve grid resilience.

Power Grid Congestion Management via Topology Optimization with AlphaZero

The paper "Power Grid Congestion Management via Topology Optimization with AlphaZero" (2211.05612) presents an innovative approach to managing power grid congestion. Utilizing AlphaZero, a modified reinforcement learning framework, the study aims to reduce reliance on costly traditional methods such as redispatching and curtailment, which are often detrimental to both operational costs and environmental goals. Below is a detailed examination of the methods, experimental results, and implications outlined in the paper.

Introduction to Power Grid Congestion Management

Modern power grids face significant challenges as renewable energy sources, such as wind and solar, become more prevalent. These sources introduce volatility that can lead to grid congestion, necessitating frequent interventions by operators to stabilize the infrastructure. Traditionally, redispatching fossil-fuel-based generators and curtailment of renewable sources have been employed as remedial actions. However, these methods are not cost-effective and counteract decarbonization efforts.

The approach proposed in the paper involves an AlphaZero-based agent that optimizes the grid topology. This method aims to mitigate congestion through non-costly measures by reconfiguring the connection topology of substations and transmission lines rather than modifying generation outputs.

Environment Design and Dynamics

The grid model is framed as an undirected multigraph where nodes represent substations and edges represent transmission lines. The operational challenge includes maintaining grid stability against events such as unexpected load spikes or generation variability.

The study leverages the Grid2Op framework to simulate the power grid environment, providing a comprehensive platform that encapsulates the complexities of grid operations, including generator and load states, and power line conditions.

Figure 1: Example topology action illustrating node splitting to resolve overflow.

AlphaZero-Based Topology Optimization

The core contribution of the paper is the integration of AlphaZero into the topology optimization problem. AlphaZero, originally developed for two-player games like Go and Chess, has been adapted to single-player grid management scenarios using a tailored Monte Carlo Tree Search (MCTS).

During simulations, the algorithm's PUCT strategy guides exploration within the tree, selecting actions based on visitation counts, policy network probabilities, and heuristic value functions—a deviation from standard neural network-based value predictions.

Figure 2: Example of a grid topology MCTS tree with critical, black-out, and skipped states.

Experimental Evaluation

The experimental setup involves benchmarking the AlphaZero-powered agent against several baselines, notably traditional redispatch and brute-force topology search. The topology optimization agent demonstrated superior performance, improving survival rates and reducing redispatch interventions, which confirms the interoperability with traditional congestion management methods.

Figures reveal promising results, particularly from the combined agent that integrates topology optimization with redispatching, showcasing a substantial reduction in operational costs. The agent's ability to maintain grid stability longer in simulation environments indicates its potential effectiveness in real-world applications.

Figure 3: Evolution of the topology agent during training.

Practical Implications and Future Directions

The study highlights the practical relevance of integrating RL-based topology optimization into day-ahead planning and real-time remedial actions in power grid operations, suggesting feasible applications in human-in-the-loop scenarios for enhanced grid reliability.

Figure 4: The WCCI 2022 power grid including all relevant components.

Two key avenues for future research include enhanced integration of redispatching during topology optimization agent training and improved action space representations. Effective solutions for these challenges could further potentiate the deployment of AlphaZero-based systems in real-world grid management and other large-scale combinatorial problems.

Conclusions

This work presents a compelling case for AlphaZero-based topology optimization as a tool for congestion management in electrical power grids. By reducing reliance on redispatching and improving grid resilience, the proposed method aligns with global decarbonization goals, offering a pathway to more sustainable and efficient energy management.

Figure 5: Overview of AlphaZero-powered real-time remedial action recommendation assistant.