Papers
Topics
Authors
Recent
Search
2000 character limit reached

ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolution

Published 2 Feb 2024 in cs.NE and cs.AI | (2402.01145v3)

Abstract: The omnipresence of NP-hard combinatorial optimization problems (COPs) compels domain experts to engage in trial-and-error heuristic design. The long-standing endeavor of design automation has gained new momentum with the rise of LLMs. This paper introduces Language Hyper-Heuristics (LHHs), an emerging variant of Hyper-Heuristics that leverages LLMs for heuristic generation, featuring minimal manual intervention and open-ended heuristic spaces. To empower LHHs, we present Reflective Evolution (ReEvo), a novel integration of evolutionary search for efficiently exploring the heuristic space, and LLM reflections to provide verbal gradients within the space. Across five heterogeneous algorithmic types, six different COPs, and both white-box and black-box views of COPs, ReEvo yields state-of-the-art and competitive meta-heuristics, evolutionary algorithms, heuristics, and neural solvers, while being more sample-efficient than prior LHHs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. Knowledge-guided local search for the vehicle routing problem. Computers & Operations Research, 105:32–46, 2019.
  2. Machine learning for combinatorial optimization: a methodological tour d’horizon. European Journal of Operational Research, 290(2):405–421, 2021.
  3. RL4CO: a unified reinforcement learning for combinatorial optimization library. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning, 2023.
  4. Large language models can implement policy iteration. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  5. A dynamic space reduction ant colony optimization for capacitated vehicle routing problem. Soft Computing, 26(17):8745–8756, 2022.
  6. A match made in consistency heaven: when large language models meet evolutionary algorithms, 2024.
  7. Evoprompting: Language models for code-level neural architecture search. arXiv preprint arXiv:2302.14838, 2023a.
  8. Efficient meta neural heuristic for multi-objective combinatorial optimization. In Advances in Neural Information Processing Systems, 2023b.
  9. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128, 2023c.
  10. Recent advances in selection hyper-heuristics. European Journal of Operational Research, 285(2):405–428, 2020.
  11. Bq-nco: Bisimulation quotienting for efficient neural combinatorial optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  12. A gp hyper-heuristic approach for generating tsp heuristics. In 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp.  521–529. IEEE, 2019.
  13. Fidanova, S. Hybrid ant colony optimization algorithm for multiple knapsack problem. In 2020 5th IEEE International Conference on Recent Advances and Innovations in Engineering (ICRAIE), pp.  1–5. IEEE, 2020.
  14. Towards generalizable neural solvers for vehicle routing problems via ensemble with transferrable local policy. arXiv preprint arXiv:2308.14104, 2023.
  15. A hierarchical hyper-heuristic for the bin packing problem. Soft Computing, 27(18):12997–13010, 2023.
  16. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532, 2023.
  17. Evolving code with a large language model, 2024.
  18. Hordijk, W. A measure of landscapes. Evolutionary computation, 4(4):335–360, 1996.
  19. Hromkovič, J. Algorithmics for hard problems: introduction to combinatorial optimization, randomization, approximation, and heuristics. Springer Science & Business Media, 2013.
  20. Graph neural network guided local search for the traveling salesperson problem. In International Conference on Learning Representations, 2022.
  21. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023.
  22. Ensemble-based deep reinforcement learning for vehicle routing problems under distribution shift. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  23. Pointerformer: Deep reinforced multi-pointer transformer for the traveling salesman problem. arXiv preprint arXiv:2304.09407, 2023.
  24. Learning the travelling salesperson problem requires rethinking generalization. arXiv preprint arXiv:2006.07054, 2020.
  25. Sym-nco: Leveraging symmetricity for neural combinatorial optimization. Advances in Neural Information Processing Systems, 35:1936–1949, 2022.
  26. Local search gflownets. arXiv preprint arXiv:2310.02710, 2023.
  27. Ant colony optimization and local search for bin packing and cutting stock problems. Journal of the Operational Research society, 55(7):705–716, 2004.
  28. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey, 2024.
  29. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  30. T2t: From distribution learning in training to gradient search in testing for combinatorial optimization. In Advances in Neural Information Processing Systems, 2023.
  31. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.  9493–9500. IEEE, 2023.
  32. Simulated-annealing-based hyper-heuristic for flexible job-shop scheduling. Engineering Optimization, 55(10):1635–1651, 2023.
  33. Large language model for multi-objective evolutionary optimization. arXiv preprint arXiv:2310.12541, 2023a.
  34. Algorithm evolution using large language model. arXiv preprint arXiv:2311.15249, 2023b.
  35. An example of evolutionary computation + large language model beating human: Design of efficient guided local search, 2024.
  36. Large language models as evolutionary optimizers. arXiv preprint arXiv:2310.19046, 2023c.
  37. How good is neural combinatorial optimization? a systematic evaluation on the traveling salesman problem. IEEE Computational Intelligence Magazine, 18(3):14–28, 2023d.
  38. Fully autonomous programming with large language models. arXiv preprint arXiv:2304.10423, 2023.
  39. Medkpl: a heterogeneous knowledge enhanced prompt learning framework for transferable diagnosis. Journal of Biomedical Informatics, pp.  104417, 2023.
  40. Neural combinatorial optimization with heavy decoder: Toward large scale generalization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  41. Learning to search feasible and infeasible regions of routing problems with flexible neural k-opt. In Advances in Neural Information Processing Systems, 2023a.
  42. Eureka: Human-level reward design via coding large language models. arXiv preprint arXiv:2310.12931, 2023b.
  43. Metabox: A benchmark platform for meta-black-box optimization with reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, 2023c.
  44. Learning performance-improving code edits. arXiv preprint arXiv:2302.07867, 2023a.
  45. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023b.
  46. A new hyper-heuristic based on ant lion optimizer and tabu search algorithm for replica management in cloud environment. Artificial Intelligence Review, 56(9):9837–9947, 2023.
  47. Analyzing the landscape of a graph based hyper-heuristic for timetabling problems. In Proceedings of the 11th Annual conference on Genetic and evolutionary computation, pp.  341–348, 2009.
  48. Hyper-heuristics: theory and applications. Springer, 2018.
  49. Automatic prompt optimization with” gradient descent” and beam search. arXiv preprint arXiv:2305.03495, 2023.
  50. Mathematical discoveries from program search with large language models. Nature, pp.  1–3, 2023.
  51. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  52. Skinderowicz, R. Improving ant colony optimization efficiency for solving large tsp instances. Applied Soft Computing, 120:108653, 2022.
  53. Acs-ophs: Ant colony system for the orienteering problem with hotel selection. EURO Journal on Transportation and Logistics, 10:100036, 2021.
  54. Solving np-hard min-max routing problems as sequential generation with equity context. arXiv preprint arXiv:2306.02689, 2023.
  55. Neuralgls: learning to guide local search with graph convolutional network for the traveling salesman problem. Neural Computing and Applications, pp.  1–20, 2023.
  56. Difusco: Graph-based diffusion solvers for combinatorial optimization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  57. Guided local search and its application to the traveling salesman problem. European Journal of Operational Research, 113(2):469–499, March 1999. ISSN 03772217. doi: 10.1016/S0377-2217(98)00099-X.
  58. Asp: Learn a universal neural solver! IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
  59. Can language models solve graph problems in natural language? arXiv preprint arXiv:2305.10037, 2023a.
  60. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint arXiv:2310.01361, 2023b.
  61. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432, 2023c.
  62. Promptagent: Strategic planning with language models enables expert-level prompt optimization. arXiv preprint arXiv:2310.16427, 2023d.
  63. Drdt: Dynamic reflection with divergent thinking for llm-based sequential recommendation. arXiv preprint arXiv:2312.11336, 2023e.
  64. Weinberger, E. Correlated and uncorrelated fitness landscapes and how to tell the difference. Biological cybernetics, 63(5):325–336, 1990.
  65. Evolutionary computation in the era of large language model: Survey and roadmap, 2024.
  66. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864, 2023.
  67. Reinforcement learning-based non-autoregressive solver for traveling salesman problems. arXiv preprint arXiv:2308.00560, 2023.
  68. Distilling autoregressive models to obtain high-performance non-autoregressive solvers for vehicle routing problems with faster inference speed. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
  69. Large language models as optimizers. arXiv preprint arXiv:2309.03409, 2023a.
  70. Intercode: Standardizing and benchmarking interactive coding with execution feedback. arXiv preprint arXiv:2306.14898, 2023b.
  71. A survey on reinforcement learning for combinatorial optimization. In 2023 IEEE World Conference on Applied Intelligence and Computing (AIC), pp.  131–136. IEEE, 2023.
  72. Deepaco: Neural-enhanced ant systems for combinatorial optimization. In Advances in Neural Information Processing Systems, 2023.
  73. Glop: Learning global partition and local construction for solving large-scale routing problems in real-time. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
  74. Automatic hyper-heuristic to generate heuristic-based adaptive sliding mode controller tuners for buck-boost converters. In Proceedings of the Genetic and Evolutionary Computation Conference, pp.  1482–1489, 2023.
  75. Self-taught optimizer (stop): Recursively self-improving code generation. arXiv preprint arXiv:2310.02304, 2023.
  76. Proagent: Building proactive cooperative ai with la rge language models. arXiv preprint arXiv:2308.11339, 2023a.
  77. Planning with large language models for code generation. In The Eleventh International Conference on Learning Representations, 2022.
  78. A q-learning-based hyper-heuristic evolutionary algorithm for the distributed flexible job-shop scheduling problem with crane transportation. Expert Systems with Applications, 234:121050, 2023b.
  79. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  80. Learning large neighborhood search for vehicle routing in airport ground handling. IEEE Transactions on Knowledge and Data Engineering, 2023.
  81. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
  82. Surgical cases assignment problem using an efficient genetic programming hyper-heuristic. Computers & Industrial Engineering, 178:109102, 2023.
Citations (21)
View on Semantic Scholar

Summary

  • The paper presents a novel framework that integrates LLM-driven reflective evolution with evolutionary search to generate diverse heuristics for NP-hard optimization problems.
  • It introduces a unique verbal gradient mechanism that refines heuristic selection and crossover across classic problems like TSP, CVRP, and BPP.
  • Empirical results reveal that ReEvo consistently achieves state-of-the-art or competitive performance, challenging traditional fixed heuristic designs in optimization.

An Expert Analysis on "LLMs as Hyper-Heuristics for Combinatorial Optimization"

The paper presents a compelling exploration into leveraging LLMs as hyper-heuristics (HHs) for solving combinatorial optimization problems (COPs), a field characterized by inherently challenging NP-hard problems. It proposes a novel framework called Language Hyper-Heuristics (LHHs), which combines LLMs' capacity to generate diverse and effective heuristics with a unique evolutionary algorithm, Reflective Evolution (ReEvo), extending the traditional boundaries of hyper-heuristics.

Methodological Advancements

ReEvo presents an integration of evolutionary search mechanisms with LLM reflections, offering what the authors term "verbal gradients" to enhance heuristic search. This method encompasses several steps: initializing a population of heuristics, conducting selection and crossover, and utilizing reflection mechanisms for refinement. Reflection involves a comparative analysis of heuristics, wherein LLMs provide insight by interpreting the relative performance between heuristics, akin to human expert feedback. This recursive refinement via evolutionary techniques marks a departure from static heuristic spaces defined by human preconceptions, broadening the possibilities for solution exploration.

Empirical Evaluation

The paper provides an extensive empirical analysis across five algorithm types and six diverse COPs, including classic problems such as Traveling Salesman (TSP), Capacitated Vehicle Routing (CVRP), and Bin Packing (BPP). Evaluation metrics demonstrate that ReEvo-attained heuristics consistently yield state-of-the-art or competitive results compared to human-designed and neural-enhanced algorithms. Notable results include outperforming the Guided Local Search (GLS) methods with penalties fine-tuned by ReEvo, thereby underscoring the model’s strength in heuristic innovation.

Theoretical and Practical Implications

This study implicitly challenges the prevailing reliance on fixed heuristic primitives by showcasing LLMs' capability to flexibly innovate in heuristic design. The implications span both theoretical landscapes, such as fostering smoother fitness landscapes and enhanced sample efficiency in searching heuristic spaces, and practical realms, particularly for black-box optimization tasks prevalent in industrial applications. The method’s adaptability across disparate algorithms and problem scales underscores its robustness, extending applicability to novel, real-world problems beyond canonical benchmarks.

Challenges and Future Directions

While the results are promising, efficient evaluation in computationally intensive scenarios remains a limitation, highlighting the trade-offs between heuristic search scope and evaluation costs. Additionally, reliance on proprietary LLMs like GPT-3.5 poses cost and accessibility issues. Future research might focus on expanding this methodology to include open-source LLMs or hybrid models to democratize access and further diversify heuristic generation across domains.

The evolution from heuristic design rigidity towards dynamically generated, scalable strategies represents a substantive advancement in solving COPs. This paper contributes a significant theoretical framework and set of empirical validations that may inspire further explorations into automated heuristic optimization using AI-driven methodologies.

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up