LLM4ED: Large Language Models for Automatic Equation Discovery

Abstract: Equation discovery is aimed at directly extracting physical laws from data and has emerged as a pivotal research domain. Previous methods based on symbolic mathematics have achieved substantial advancements, but often require the design of implementation of complex algorithms. In this paper, we introduce a new framework that utilizes natural language-based prompts to guide LLMs in automatically mining governing equations from data. Specifically, we first utilize the generation capability of LLMs to generate diverse equations in string form, and then evaluate the generated equations based on observations. In the optimization phase, we propose two alternately iterated strategies to optimize generated equations collaboratively. The first strategy is to take LLMs as a black-box optimizer and achieve equation self-improvement based on historical samples and their performance. The second strategy is to instruct LLMs to perform evolutionary operators for global search. Experiments are extensively conducted on both partial differential equations and ordinary differential equations. Results demonstrate that our framework can discover effective equations to reveal the underlying physical laws under various nonlinear dynamic systems. Further comparisons are made with state-of-the-art models, demonstrating good stability and usability. Our framework substantially lowers the barriers to learning and applying equation discovery techniques, demonstrating the application potential of LLMs in the field of knowledge discovery.

• The paper presents a novel framework that leverages LLMs to automatically generate and optimize governing equations from observed data.
• It combines self-improvement and evolutionary strategies to iteratively refine candidate equations using symbolic math tools and optimization methods.
• The approach reduces reliance on complex algorithms and prior domain knowledge, achieving promising accuracy on nonlinear PDEs and ODEs.

LLM4ED: LLMs for Automatic Equation Discovery

This paper presents a framework leveraging LLMs to automatically discover governing equations from observed data. Traditionally, symbolic mathematics methods have dominated this domain but often involve complex algorithmic design and are reliant on prior domain knowledge. This new approach utilizes LLMs to generate and optimize mathematical equations using natural language prompts, bypassing the need for intricate algorithm implementations.

Proposed Framework

The core idea behind the LLM4ED framework involves guiding LLMs using prompts to perform two main tasks: generation and optimization of equation candidates. The framework consists of the following stages:

1. Equation Generation:
   • LLMs generate a diverse set of equations in string format from a defined symbol library and problem description.
2. Optimization:
   • Two strategies are used: self-improvement and evolutionary operations.
   • Self-Improvement: Treats LLMs as black-box optimizers, iteratively refining equations based on historical performance data.
   • Evolutionary Search: Employs LLMs to perform genetic algorithm-inspired operations like crossover and mutation to explore globally.

     Figure 1: Overview of the proposed framework.

     

     Figure 2: Workflow of the proposed framework.

Methodology

Initialization

Equations are generated via LLMs using predefined symbol libraries. This initial generation forms a starting population for evaluation and optimization.

Evaluation

Equations are parsed using symbolic math tools (e.g., Sympy) to convert string formats into expression trees. Then, constants within these expressions are determined through optimization methods such as sparse regression for PDEs and BFGS for ODEs. The quality of each equation is rated using a scoring function that combines accuracy of fit and expression complexity.

Optimization Strategies

• Self-Improvement:

Utilizes previous high-scoring equations to guide LLMs in crafting refined versions through local modifications, helping to remove redundancies and introduce new variations.

Figure 3: Self-improvement process executed by LLMs.

• Genetic Algorithms:

LLMs conduct global searches by applying genetic operators such as crossover and mutation on a population of high-quality equations.

Figure 4: Crossover and mutation executed by LLMs.

Experimental Results

Evaluation of the framework was performed on various nonlinear dynamic system equations, notably PDEs and ODEs. The results indicate strong performance with low coefficient errors across multiple scenarios, such as Burgers', Chafee-Infante, and Kuramoto-Sivashinsky equations. Additionally, comparisons with state-of-the-art models showed that the LLM4ED framework provides comparable or superior performance in terms of stability and usability.

Figure 5: Discovered results under different optimization methods.

Practical Implications and Future Work

The framework is a breakthrough in reducing the learning curve and operational complexity associated with equation discovery. By automating the generation and optimization of equations, it broadens access to these techniques beyond expert domains. Future iterations of the framework could integrate more sophisticated prompts to refine its search space further and apply streamlined evaluation criteria for more complex systems with sparse or noisy data.

Conclusion

LLM4ED has shown its potential to transform the field of automated equation discovery by effectively combining the generative and reasoning capabilities of LLMs. This paper concludes with the promising application prospects of LLMs in scientific knowledge discovery, encouraging further exploration and refinement of this AI-driven methodology.