Exploring and Improving the Spatial Reasoning Abilities of Large Language Models

Abstract: LLMs represent formidable tools for sequence modeling, boasting an innate capacity for general pattern recognition. Nevertheless, their broader spatial reasoning capabilities, especially applied to numerical trajectory data, remain insufficiently explored. In this paper, we investigate the out-of-the-box performance of ChatGPT-3.5, ChatGPT-4 and Llama 2 7B models when confronted with 3D robotic trajectory data from the CALVIN baseline and associated tasks, including 2D directional and shape labeling. Additionally, we introduce a novel prefix-based prompting mechanism, which yields a 33% improvement on the 3D trajectory data and an increase of up to 10% on SpartQA tasks over zero-shot prompting (with gains for other prompting types as well). The experimentation with 3D trajectory data offers an intriguing glimpse into the manner in which LLMs engage with numerical and spatial information, thus laying a solid foundation for the identification of target areas for future enhancements.

• The paper demonstrates that spatial prefix-prompting significantly improves LLMs’ performance on both 2D path labeling and 3D trajectory tasks.
• It employs methods such as zero-shot, in-context, and chain-of-thought prompting, with ChatGPT-4 achieving perfect accuracy on short 2D trajectories.
• Findings reveal a performance drop on complex 3D tasks, highlighting the need for further research in enhancing spatial reasoning in large language models.

Background and Objectives

LLMs have demonstrated impressive abilities in extrapolating patterns and serving as tools for cross-disciplinary applications. Despite these capabilities, their proficiency in more abstract areas, such as spatial reasoning, is less well-understood. This study aims to assess the performance of LLMs, specifically ChatGPT versions 3.5 and 4, and Llama 2 7B, in tasks requiring spatial understanding. These tasks involve labeling 2D paths and identifying shapes, as well as labeling 3D robotic trajectories.

Approach and Methodology

To investigate these capabilities, the study generates datasets for 2D path and shape labeling, using simple directional instructions and shapes like circles. For 3D trajectory labeling, it employs the CALVIN baseline, which contains data on robotic movements. The researchers evaluate the models using zero-shot prompting, In-context Learning (ICL), Chain-of-Thought (CoT) prompting, and propose a new method, Spatial Prefix-Prompting (SPP), which introduces a related spatial problem before the primary query. The study examines not only how LLMs perform with simple spatial patterns but also the transfer of knowledge from simpler tasks to more complex ones.

Results and Findings

The experiments reveal that LLMs are competent at identifying simple 2D spatial patterns and yield acceptable few-shot identification of directions, especially with ChatGPT-4, which reaches perfect classification rates on short trajectories. However, the performance drops significantly when dealing with more complex 3D trajectories, with even the best models achieving only 80% accuracy after employing SPP on the "cleaned" CALVIN dataset where noise is reduced. CoT prompting showed inconsistent performance and did not always yield improvements, suggesting it may not be as effective for spatial tasks compared to language or mathematical reasoning.

Implications and Future Directions

The Spatial Prefix-Prompting method showed promise, often outperforming other techniques, which indicates that prompting models with simpler, related problems can facilitate better performance on complex spatial tasks. This study lays the groundwork for future research into enhancing the spatial reasoning abilities of LLMs. Potential applications could extend to areas such as trend analysis or time-series data interpretation. Going forward, the research could benefit from a larger dataset and exploring additional spatial tasks including 3D point-cloud analysis and multi-variable trend forecasting.