Adaptive Selection of Symbolic Languages for Improving LLM Logical Reasoning

Abstract: LLMs still struggle with complex logical reasoning. While previous works achieve remarkable improvements, their performance is highly dependent on the correctness of translating natural language (NL) problems into a symbolic language (SL). Though numerous works focusing on improving this translation accuracy, they only consider the similarity between the meaning of SL and NL, overlooking another crucial influencing factor, the selection of the target SL type itself. For example, first-order logic language specializes in logical reasoning with categorical syllogisms and complex quantifiers, while Boolean satisfiability formalism excels at representing constraint satisfaction like partial problems. To our knowledge, this is the first paper to claim and verify that different NL logical reasoning problem corresponds to different optimal SL formalization for translation. Based on this, we propose a methods to improve the logical reasoning performance of LLMs by adaptively selecting the most suitable SL for each problem prior to translation. Specifically, we leverage LLMs to select the target SL among first-order logic, logic programming and Boolean satisfiability and then translate the problem in NL to target SL expressions as well as employ the corresponding logical solver to derive the final answer. Experimental results on benchmarks show that our adaptive selection method significantly outperforms translating all into single SL and randomly selecting the SL. On a mixed dataset of these benchmarks, our approach achieves 96% accuracy, which improving performance by 25% compared to the second highest accuracy from the first-order logic translation.

• The paper introduces an adaptive selection algorithm that chooses the optimal symbolic language based on the natural language problem's structure.
• The methodology dynamically translates problems into symbolic form using solvers like Prover9, Pyke, and Z3, achieving 96.00% accuracy on mixed datasets.
• The adaptive approach outperforms static and random selection techniques, paving the way for more advanced neuro-symbolic reasoning systems.

Adaptive Selection of Symbolic Languages for Improving LLM Logical Reasoning

Introduction

This paper investigates the capability of LLMs to perform logical reasoning tasks via an innovative method of adaptive symbolic language (SL) selection. While previous efforts have improved LLM performance in logic-heavy tasks by translating natural language (NL) problems into symbolic languages, such efforts are contingent on the translation's accuracy. Notably, the choice of SL is crucial, as different languages have distinct strengths depending on the problem's logical structure. The research demonstrates the benefit of aligning the SL's characteristics with NL problem attributes to enhance translation precision and logical inference accuracy.

Methodology

The methodology revolves around dynamically selecting the most suitable SL for logical reasoning tasks among first-order logic (FOL), logic programming (LP), and Boolean satisfiability (SAT). The approach comprises three main stages:

1. Adaptive SL Selection: This initial stage involves using LLMs to evaluate the NL problem's structure and select the optimal SL. The model examines expressive features of possible SLs and assesses which aligns best with the problem's logical requirements.

   Figure 1: The framework of our methods to adaptively select symbolic languages to translate logical reasoning problems.

2. Translation: Once the SL is selected, the LLM translates the NL description into the symbolic formulation in the chosen SL. By doing so, the LLM ensures that the translation captures the NL problem's semantics fully and accurately.
3. Reasoning via Solvers: The translated problem is then tackled by a specialized logic solver appropriate for the chosen SL. The solvers used are Prover9 for FOL, Pyke for LP, and Z3 for SAT, each providing tailored inference capabilities.

Results and Analysis

The experiments across multiple benchmarks demonstrate the effectiveness of the adaptive SL selection. The proposed method achieved impressive accuracy gains, particularly notable on mixed datasets where it reached 96.00% accuracy, showcasing a substantial improvement over methods using a single SL approach.

• Performance Metrics: The method consistently outperformed existing techniques, with substantial improvements in both execution rates and accuracy on diverse datasets such as ProntoQA, ProofWriter, and LogicalDeduction.
• Comparison to Baselines: The adaptive selection strategy significantly outperformed a static SL choice and random selection, reinforcing the necessity of dynamic SL alignment to leverage LLMs' reasoning abilities fully.

Implications and Future Work

This research highlights a critical dimension in the interplay between natural language processing and symbolic logic reasoning. It underscores the importance of selecting the appropriate SL as a fundamental step for improving LLM performance in logical reasoning tasks. By facilitating precise translations and leveraging specialized solvers, the method reinforces the emergent direction of hybrid neuro-symbolic AI systems.

Future research could explore integrating additional SLs, refining the adaptive selection mechanics further, and extending the framework to encompass more complex reasoning scenarios, thereby broadening the applicability and robustness of LLM-based reasoning frameworks. Additionally, the establishment of a formal theoretical foundation could guide SL selections and improve automated reasoning processes in complex multi-domain environments.

Conclusion

The paper presents a significant advance in addressing the translation and reasoning limitations inherent in LLMs faced with logic-intensive tasks. By implementing an adaptive selection algorithm for symbolic languages, it significantly enhances translation accuracy and reasoning capability, demonstrating notable performance increases across various logical reasoning benchmarks. The insights gained pave the way for more sophisticated neuro-symbolic systems, pointing to promising avenues for future AI research and applications.