Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective

Abstract: Graphs are a natural representation for systems based on relations between connected entities. Combinatorial optimization problems, which arise when considering an objective function related to a process of interest on discrete structures, are often challenging due to the rapid growth of the solution space. The trial-and-error paradigm of Reinforcement Learning has recently emerged as a promising alternative to traditional methods, such as exact algorithms and (meta)heuristics, for discovering better decision-making strategies in a variety of disciplines including chemistry, computer science, and statistics. Despite the fact that they arose in markedly different fields, these techniques share significant commonalities. Therefore, we set out to synthesize this work in a unifying perspective that we term Graph Reinforcement Learning, interpreting it as a constructive decision-making method for graph problems. After covering the relevant technical background, we review works along the dividing line of whether the goal is to optimize graph structure given a process of interest, or to optimize the outcome of the process itself under fixed graph structure. Finally, we discuss the common challenges facing the field and open research questions. In contrast with other surveys, the present work focuses on non-canonical graph problems for which performant algorithms are typically not known and Reinforcement Learning is able to provide efficient and effective solutions.

• The paper presents a unifying perspective on applying reinforcement learning to combinatorial optimization on graphs via a Markov Decision Process framework.
• It categorizes methodologies into Graph Structure Optimization, which alters graph topologies, and Graph Process Optimization, which refines control actions on fixed graphs.
• The study highlights challenges like scalability, generalization, and interpretability, paving the way for hybrid RL models and future research directions.

Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective

Introduction

The paper "Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective" (2404.06492) provides a comprehensive examination of the application of Reinforcement Learning (RL) techniques to combinatorial optimization problems on graph structures. The authors aim to unify disparate approaches from various fields under the term Graph Reinforcement Learning (Graph RL), emphasizing its utility in addressing complex decision-making challenges that are not easily tackled by traditional methods.

Graphs provide a natural representation for systems where entities are connected by relationships, which are prevalent in domains such as chemistry, computer science, and the social sciences. Combinatorial optimization on graphs involves finding a solution that maximizes or minimizes an objective function defined over these structures. The solution space for such problems grows rapidly, making them computationally challenging.

Technical Background and Methodologies

The paper begins by elucidating the foundational aspects of graphs and combinatorial optimization. It proceeds to discuss how RL, particularly through the Markov Decision Process (MDP) framework, offers a viable alternative to classical approaches such as exact algorithms and (meta)heuristics. RL's trial-and-error paradigm, which enables the automatic discovery of decision-making strategies, is particularly advantageous for non-canonical graph problems where established algorithms are inadequate.

Graph RL problems are categorized into two primary types: Graph Structure Optimization and Graph Process Optimization. Graph Structure Optimization focuses on modifying the topology of a graph to improve an objective, while Graph Process Optimization involves optimizing outcomes of processes defined over a fixed graph structure.

Figure 1: Visual summary of the structure and topics of the present survey.

Graph Structure Optimization

In Graph Structure Optimization, the focus is on altering the graph’s topology—through edge additions, removals, or rewiring—to enhance an objective function. This approach is relevant in various contexts, including:

• Attacking Graph Neural Networks: Modifying graph structures to induce GNNs to make classification errors [dai_adversarial_2018].
• Network Design: Constructing or modifying network topologies to improve resilience, efficiency, or other network metrics [darvariu2021goal].
• Causal Discovery: Identifying causal relationships by constraining graph structures to be acyclic and maximizing explanatory power [zhu2020causal].
• Molecular Optimization: Devising molecular structures with optimal properties, such as drug-likeness [you_graph_2018].

  Figure 2: High-level illustration of how Graph Structure Optimization problems are approached with RL.

Graph Process Optimization

Graph Process Optimization is concerned with optimizing a process over a static graph structure by choosing the best set of control actions. Significant applications include:

• Routing on Networks: Optimizing flow distribution in networks to minimize congestion or maximize throughput [valadarsky2017learning].
• Network Games: Identifying equilibrium states in network-based games that optimize social welfare or fairness [darvariu2021solvingshort].
• Spreading Processes: Controlling processes such as epidemic spreading to minimize infections by optimally selecting influenced nodes [meirom2021controlling].
• Search and Navigation: Developing strategies for effective exploration and pathfinding in graph structures, with applications in knowledge graph completion and robotic motion planning [shenMWalkLearningWalk2018].

  Figure 3: High-level illustration of how Graph Process Optimization problems are approached with RL.

Challenges and Implications

The paper outlines several key challenges in applying Graph RL:

• Scalability: Managing the computational demands of large-scale graph problems.
• Generalization: Ensuring models perform well across various graph instances and do not overfit to specific scenarios.
• Interpretability: Extracting human-interpretable insights from learned RL models to enhance understanding and further optimization.

Addressing these challenges involves developing more efficient RL algorithms, leveraging domain-specific insights, and integrating RL with hybrid models that combine traditional algorithms with machine learning techniques.

Conclusion

Graph RL emerges as a powerful framework for tackling combinatorial optimization problems that are intractable with classical methods. By synthesizing diverse approaches under a unifying perspective, this work sets the stage for further advancements in both the theory and application of Graph RL. The implications extend across numerous domains, offering new paradigms for optimizing complex systems structured as graphs. As RL techniques continue to evolve, their integration with graph-based models promises significant breakthroughs in addressing both longstanding and emerging optimization challenges.