RL4CO: an Extensive Reinforcement Learning for Combinatorial Optimization Benchmark

Abstract: Combinatorial optimization (CO) is fundamental to several real-world applications, from logistics and scheduling to hardware design and resource allocation. Deep reinforcement learning (RL) has recently shown significant benefits in solving CO problems, reducing reliance on domain expertise and improving computational efficiency. However, the absence of a unified benchmarking framework leads to inconsistent evaluations, limits reproducibility, and increases engineering overhead, raising barriers to adoption for new researchers. To address these challenges, we introduce RL4CO, a unified and extensive benchmark with in-depth library coverage of 27 CO problem environments and 23 state-of-the-art baselines. Built on efficient software libraries and best practices in implementation, RL4CO features modularized implementation and flexible configurations of diverse environments, policy architectures, RL algorithms, and utilities with extensive documentation. RL4CO helps researchers build on existing successes while exploring and developing their own designs, facilitating the entire research process by decoupling science from heavy engineering. We finally provide extensive benchmark studies to inspire new insights and future work. RL4CO has already attracted numerous researchers in the community and is open-sourced at https://github.com/ai4co/rl4co.

• The paper presents RL4CO as a unified framework that benchmarks 27 combinatorial optimization environments with 23 state-of-the-art baselines.
• The methodology leverages advanced RL algorithms like PPO and REINFORCE combined with modular policy designs and efficient multi-device training.
• The benchmark demonstrates strong generalization and adaptability across diverse CO problems including TSP, CVRP, and PDP.

"RL4CO: An Extensive Reinforcement Learning for Combinatorial Optimization Benchmark"

Introduction

The paper "RL4CO: An Extensive Reinforcement Learning for Combinatorial Optimization Benchmark" addresses the challenges in combinatorial optimization (CO) using deep reinforcement learning (RL). The main obstacle in CO is the exponential complexity and NP-hard nature of these problems, which has traditionally been tackled through mathematical programming and heuristics. However, these methods often fall short in scalability and require significant domain-specific knowledge.

RL4CO introduces a comprehensive framework designed to unify and benchmark 27 CO environments with 23 state-of-the-art baselines, all leveraging the advantages of modular architectures and efficient training pipelines.

Figure 1: Overview of the RL4CO pipeline: from configurations to training a policy on an environment.

RL4CO Framework

Policy Modularization

Policies in RL4CO are divided into constructive and improvement types. Constructive policies are further categorized into autoregressive (AR) and non-autoregressive (NAR), which sequentially or globally build solutions from scratch. Improvement policies refine existing solutions either iteratively or through hybrid methods with constructive approaches.

Figure 2: Overview of different types of policies and their modularization in RL4CO.

Training Algorithms

RL algorithms aim to maximize expected cumulative rewards for CO instances without labeled data requirements. Techniques such as PPO, A2C, and variations of REINFORCE provide a robust framework for optimizing neural combinatorial optimizers.

The training infrastructure uses advanced libraries, including TorchRL and PyTorch Lightning, to enable flexible multi-device training with high resource efficiency. This setup is critical to managing the computational load of training large-scale models over diversified environments.

Benchmark Studies

Evaluation Metrics

The effectiveness of RL4CO is measured using various benchmarks tasks like TSP, CVRP, PDP, and more, emphasizing metrics such as solution quality, convergence properties, and generalization to different environments.

Figure 3: Study of decoding schemes using POMO on CVRP50. [Left]: Pareto front of decoding schemes by the number of samples; [Right]: sampling performance with different temperatures $\tau$ and $p$ values for top-p sampling.

Generalization and Scalability

RL4CO demonstrates significant generalization capabilities by training models on various VRP attributes. Models like MTPOMO exhibit strong performance across different problem variants and distributions, highlighting the potential for cross-task learning and adaptability to unseen tasks.

Sampling and Decoding Techniques

The paper underscores the importance of diverse decoding schemes including sampling techniques with softmax temperature scaling and top-p sampling to enhance exploration and solution diversity. This flexibility promotes improved solution quality for complex instances.

Implementation and Community Impact

The RL4CO benchmark offers extensive documentation and tutorials to foster wider adoption and extension by the research community. It encourages contributions and is accessible for new implementations that can broaden its application scope.

Conclusion

RL4CO emerges as a vital tool for researchers and practitioners in the NCO domain, streamlining the research and benchmarking process with its modular, flexible, and extensible framework. By unifying various RL methodologies and combinatorial problem environments, RL4CO not only supports reproducibility but also accelerates innovation in developing efficient, scalable, and generalizable solution strategies for complex CO problems.