GridRoute: A Benchmark for LLM-Based Route Planning with Cardinal Movement in Grid Environments

Abstract: Recent advancements in LLMs have demonstrated their potential in planning and reasoning tasks, offering a flexible alternative to classical pathfinding algorithms. However, most existing studies focus on LLMs' independent reasoning capabilities and overlook the potential synergy between LLMs and traditional algorithms. To fill this gap, we propose a comprehensive evaluation benchmark GridRoute to assess how LLMs can take advantage of traditional algorithms. We also propose a novel hybrid prompting technique called Algorithm of Thought (AoT), which introduces traditional algorithms' guidance into prompting. Our benchmark evaluates six LLMs ranging from 7B to 72B parameters across various map sizes, assessing their performance in correctness, optimality, and efficiency in grid environments with varying sizes. Our results show that AoT significantly boosts performance across all model sizes, particularly in larger or more complex environments, suggesting a promising approach to addressing path planning challenges. Our code is open-sourced at https://github.com/LinChance/GridRoute.

GridRoute: Evaluating LLM-Based Path Planning in Grid Environments

The paper "GridRoute: A Benchmark for LLM-Based Route Planning with Cardinal Movement in Grid Environments" explores the integration of LLMs with classical pathfinding algorithms, aiming to forge a synergy that capitalizes on the capabilities inherent to both approaches. It introduces a novel comprehensive benchmark, GridRoute, designed to evaluate LLMs' route planning performance under varied conditions, specifically focusing on the optimal navigation of a grid environment with cardinal movement constraints.

Research Premise and Contributions

The dominance of classical algorithms like A*, Dijkstra, and Depth-First Search (DFS) in grid-based pathfinding is well-established. These algorithms leverage explicit search heuristics to guarantee optimal solutions under specific conditions. LLMs, however, offer a promising alternative through their implicit reasoning and adaptive abilities, although existing studies primarily explore their independent reasoning capacity. This paper addresses the gap by examining the cooperative potential between LLMs and classical algorithms through the GridRoute framework.

The paper makes several significant contributions:

1. GridRoute Benchmark Creation: It introduces GridRoute, capable of systematic comparisons between various LLMs and classical algorithms in simulated grid environments. This allows for rigorous evaluations of correctness, optimality, and efficiency across different map sizes and complexities.
2. Algorithm of Thought (AoT) Prompting: A core innovation is the AoT prompting technique, which embeds traditional algorithms' guidance within the prompting framework for LLMs. This enhances an LLM's planning capabilities by combining algorithm-generated trajectories with reasoning-based prompts.
3. Extensive Experiments: The study conducts experiments using six models from different LLM families, assessing performance through metrics like Compliance Ratio, Feasibility Ratio, Optimal Ratio, Geometric Mean, Mean Square Error, and Runtime. These experiments demonstrate the improvements in planning performance when algorithmic guidance is integrated into LLM prompting.

Methodology and Findings

GridRoute's methodology utilizes a variety of prompting strategies to test the intersection of LLMs and algorithmic guidance. The study crafts independent route planning prompts, AoT prompts, and AoT with Example prompts (Algo-Shot), which collectively illustrate the impact of algorithm guidance under varied conditions.

Key findings include:

• Enhanced Performance with AoT: AoT prompts significantly outperform vanilla prompts across all evaluation metrics, demonstrating superior ability in path planning tasks, particularly in complex grid environments.
• Model Scale Impact: Larger models generally exhibit enhanced accuracy and reduced errors, yet beyond a certain scale, performance tends to plateau, highlighting diminishing returns as model complexity increases.
• Complementary Strengths: The synergy between LLMs and classical algorithms is evident, with algorithmic guidance leading to marked improvements in LLM-based pathfinding despite increased map complexity.

Implications and Future Directions

The implications of this research stretch across theoretical and practical domains. Theoretically, it enriches our understanding of how structured reasoning and algorithmic guidance can aid LLMs in complex tasks, suggesting possible expansions into hybrid neuro-symbolic systems. Practically, applications in areas such as autonomous robotics and logistics could benefit immensely from the enhanced planning capabilities demonstrated by integrating LLMs with classical algorithms.

Future research directions could explore ultra-large-scale maps, multi-goal cooperative planning scenarios, and refined algorithmic strategies to address persistent issues, like path traversal through obstacles, in LLM-based path planning. The adaptable GridRoute framework supports further extensions to evaluate the generalization of LLMs in increasingly complicated planning tasks.