Dynamic Model Discovery (LLM-DMD)

• Dynamic Model Discovery (LLM-DMD) is a framework that infers governing differential equations from time-series and high-dimensional data using large language models and numerical inversion techniques.
• It employs methodologies such as linear multistep inversion, entropic regression, and LLM-guided code synthesis to identify latent coordinates and symbolic dynamics.
• The approach automates variable selection and stability analysis, enhancing predictive accuracy in applications ranging from power systems to molecular dynamics.

Dynamic Model Discovery (LLM-DMD) describes the process of inferring governing dynamical equations or predictive operators from time-series, spatiotemporal, or high-dimensional measurement data, leveraging LLMs and related statistical or algorithmic frameworks. Techniques under the LLM-DMD family facilitate autonomous identification of system state variables, latent coordinates, and the symbolic or functional forms of underlying differential, algebraic, or iterative relations, often with minimal prior assumptions about the variable set, functional basis, or embedding. The paradigm spans traditional numerical approaches (linear multistep inversion), entropic regression for basis selection, and recent LLM-guided search in power systems, molecular dynamics, and high-dimensional scientific data domains.

1. Theoretical Foundations and Problem Formulation

At its core, dynamic model discovery seeks mappings $$\mathcal{M}: \text{observations} \mapsto (\text{structure}, \text{parameters})$$ such that the recovered models reproduce and extrapolate system evolution. The classical data-driven formulation provides a set of state samples $\{x_n\}$ at uniform or nonuniform times and asks for an unknown vector field $f:\mathbb{R}^d \to \mathbb{R}^d$ so that $\dot x = f(x)$. Multistep approaches—particularly LMM-DMD—recast the problem in terms of discrete residual minimization:

$$\sum_{j=0}^M \alpha_j x_{n-j} = h \sum_{j=0}^M \beta_j f(x_{n-j}),$$

where $\alpha_j$, $\beta_j$ are fixed scalars (coefficients from Adams–Bashforth, Adams–Moulton, or BDF families), $h$ is the timestep, and $M$ is the number of steps (Keller et al., 2019).

Modern extensions generalize to differential-algebraic equations (DAEs), symbolic hypothesis spaces, and latent coordinate identification. The framework is thus characterized by:

• Inverse modeling: Given observed state–time pairs, infer both the coordinate map $\phi: \mathbb{R}^D \to \mathbb{R}^d$ and the governing ODE, PDE, or discrete-time evolution.
• Zero-residual idealization: Assume residuals $R_n[x,f] \equiv 0$ in the noiseless case—generalize to penalized regression in the presence of measurement error or roundoff.
• Autonomous and non-autonomous formulations: Applicable to both time-invariant and periodically-forced systems, with adaptation for slow spectral submanifolds (SSMs) (Haller et al., 2024).

LLMs and multimodal LLMs (MLLMs) further introduce automatic symbolic reasoning and code synthesis, allowing model discovery in large, combinatorial search spaces, often zero-shot or with in-context reinforcement (Shen et al., 9 Jan 2026, Li et al., 17 May 2025).

2. Key Methodological Approaches

2.1 Linear Multistep Methods for Discovery (LMM-DMD)

LMM-DMD inverts the traditional numerical propagation, solving for $f(\cdot)$ rather than advancing $x(\cdot)$ (Keller et al., 2019). The discovery procedure includes:

• Forming the block system: $B f = h^{-1} A x - g$, where $B$ is a Toeplitz matrix constructed from $\beta_j$.
• Defining discrete residuals: $R_n[x, f]$ quantifies the deviation from the numerical scheme, used as a loss for optimization when $f$ is parameterized (e.g., neural networks with parameters $\theta$).
• Stability and convergence analysis: Stability is governed by the second characteristic polynomial $\omega(r) = \sum_{j=0}^M \beta_j r^{M-j}$; methods are convergent if the strong root condition (all roots strictly inside the unit disk) is satisfied.
• Scheme selection: Only BDF-M (all $M$), AB-M ($1 \le M \le 6$), and AM-M ($M=0,1$) schemes are provably stable and convergent for discovery; instability and ill-conditioning arise for AB-M ($M\ge7$) and AM-M ($M\ge2$).

2.2 Entropic Regression for Delay Selection (ERDMD)

ERDMD generalizes DMD via entropy-based pruning and construction of nonuniform time-delay models (Curtis et al., 2024). This procedure is characterized by:

• Information-theoretic lag selection: Uses conditional mutual information to identify the most informative time delays (BUILD phase), and prune those contributing no additional information (PRUNE phase).
• Sparse, interpretable models: Final delay sets $\ell_c$ typically contain only a few lags out of potentially hundreds or thousands, yielding efficient reconstructions and robust forecasting.
• Least-squares Koopman fitting: For chosen delays, finds Koopman operators $K_{l_k}$ by Frobenius-norm minimization.
• Companion matrix spectral analysis: Aggregates delay-specific Koopman matrices into a companion operator $K_a$, whose spectrum reveals timescale separation and informs model validity.

2.3 LLM-Guided Structural Equation Discovery

LLM-based DMD frameworks—such as those applied to power systems and molecular dynamics—leverage in-context code synthesis and reasoning (Shen et al., 9 Jan 2026, Murtada et al., 21 Jul 2025). Key algorithmic elements include:

• Sequential differential and algebraic equation loops: The DE loop iteratively generates code skeletons for dynamic equations, fits parameters by gradient-based optimization, scores candidates, and maintains diversity via an island-based archive.
• Variable extension: When evaluation stagnates, the framework automatically parses model requests for missing variables or inputs and augments the candidate set.
• Autoregressive or zero-shot inference: In molecular systems, trained LLMs predict structural token sequences for future states, reconstructing atomic configurations without explicit force fields; in high-dimensional systems, multimodal LLMs infer coordinate maps and governing equations in a zero-shot pipeline (Li et al., 17 May 2025).

3. Evaluation, Stability, and Convergence Conditions

Rigorous analysis distinguishes LLM-DMD workflows by their adherence to mathematical consistency, stability, and convergence:

• Consistency: Discovery-consistency is defined in terms of the truncation error $\tau_n = R_n[x,\dot x] = O(h^p)$, where $p$ is scheme order. Strong consistency entails either $\ell_\infty$ or summed residual norms decaying as $O(h^k)$ (Keller et al., 2019).
• Stability: 0-stability (inverse stability) is achieved if small data perturbations yield bounded parameter changes, with a constant $K$ independent of system size.
• Algebraic root condition: Physical convergence demands that roots of $\omega(r)$ (second characteristic polynomial) satisfy the strong root condition. Marginal stability (simple roots on the unit circle) only guarantees convergence under more stringent data and error assumptions.
• Practical matrix conditioning: Unstable schemes (ill-conditioned $B$) amplify noise; regularization and basis selection mitigate overfitting in the presence of incomplete or noisy measurements.

Empirical studies confirm theoretical predictions. In AB-M ($M\le6$) and BDF-M, error plots $\|f_h-f\|_\infty$ scale as $h^M$; unstable variants show exponential error growth or persistent residuals (Keller et al., 2019). Quantitative benchmarks (IEEE 39-bus SG system) reveal LLM-DMD models outperform SINDy-based discovery in Mean Absolute Percentage Error (MAPE) and $R^2$ score (Shen et al., 9 Jan 2026).

4. Extensions to High-Dimensional and Multimodal Systems

Multimodal LLM-DMD frameworks generalize dynamic model discovery to domains with high-dimensional and visual data (Li et al., 17 May 2025):

• Coordinate discovery: Detection toolkits and enhanced visual prompts (grid overlays, quadrant masking, plot replay) allow MLLMs to extract intrinsic coordinates from pixel data, or select latent dimensions via autoencoder bottleneck variation.
• Symbolic regression guidance: MLLMs propose candidate term libraries $\Theta(z)$ for ODE discovery, guided by prior experience and fitness–simplicity trade-offs.
• Hypothesis assessment and early stopping: Fitness pools accumulate candidate libraries and coefficients, with termination upon repeated proposals. Experience is managed in a closed-loop with iterative prompt refinement.
• Quantitative accuracy: Long-term extrapolation $R^2$ scores improve by $\approx27\%$ over baselines on video and latent system datasets.

Molecular dynamics applications (MD-LLM-1) demonstrate bidirectional state discovery—fine-tuning on one conformational basin enables prediction of transitions to others, verified by RMSD and trajectory statistics (Murtada et al., 21 Jul 2025).

5. Practical Implementation and Algorithmic Workflows

Dynamic model discovery pipelines follow a generalized sequence:

• Data acquisition: Collect sequential observations $\{x_n\}$; augment with estimated derivatives or algebraic variables if needed.
• Linear system or loss construction: Formulate the residual minimization problem, select appropriate LMM or delay schemes (based on root stability), construct parameterized representations (neural networks, symbolic libraries, or code skeletons).
• Optimization: Employ gradient-based optimizers (Adam, AdamW), annealed learning rates, regularization (code length, $L^2$ or $L^1$ penalties) to stabilize fitting.
• Model validation: Score candidate models on held-out data, exploit archive clustering for diversity, and extend variable sets in response to stagnation or missing dependencies.
• Interpretability: Analyze discovered modes, delays, or coordinate maps for physical meaning; confirm spectral properties align with system dynamics.

In entropic approaches, shuffle tests determine statistical thresholds for information gain, ensuring robustness against spurious lag selection (Curtis et al., 2024). Autoregressive prediction in molecular systems uses rolling-window inputs, tokenized via domain-specific graph encoders and quantization schemes (FoldToken, BlockGAT) (Murtada et al., 21 Jul 2025).

6. Limitations, Robustness, and Future Directions

Current LLM-DMD methods are subject to several practical and theoretical limitations:

• Reliance on LLM code synthesis: Occasional generation of syntactic or semantic errors requires filtering (Shen et al., 9 Jan 2026).
• Computational cost: Model discovery over large search spaces can be resource-intensive, especially when sampling, compiling, and optimizing numerous candidates.
• Basis dictionary completeness: Success relies on the symbolic representation capabilities of the underlying LLMs or MLLMs. Failure modes arise when the correct dynamics are outside the pretrained vocabulary.
• Prompt engineering and experience replay: Manual intervention may be needed to guide MLLM responses, particularly in visual or multimodal settings.
• Extension to PDEs, stochastic, and non-autonomous systems: Current implementations primarily target ODEs and DAEs; research on PDE operator discovery and handling of noise, non-autonomy, and non-smooth manifolds remains ongoing.

Research directions include integration with real-time data streams (phasor measurement units), extension to grid-forming converters and multimachine interactions in power systems, enforcement of thermodynamic constraints (Boltzmann-weighted loss) in molecular models, and automation of prompt refinement via closed-loop or reinforcement learning (Li et al., 17 May 2025, Murtada et al., 21 Jul 2025). On-device deployment and reduced-latency inference are also anticipated.

7. Contextual Significance and Comparative Summary

Dynamic Model Discovery (LLM-DMD) subsumes and advances several established paradigms:

Approach	Model Class	Variable Selection	Basis/Dictionary Selection	Information Criterion
LMM-DMD (Keller et al., 2019)	ODE, vector fields	Manual or fixed	LMM coefficients	Residual magnitude, root stability
ERDMD (Curtis et al., 2024)	Koopman/DMD	Automatic, sparse	Time delays (nonuniform)	Conditional mutual information
LLM-DMD (Power) (Shen et al., 9 Jan 2026)	DAEs, ODEs	LLM-autonomous, extension	Symbolic code skeleton	Gradient-based loss, archive score
MD-LLM (Murtada et al., 21 Jul 2025)	Sequence generative	Fine-tuned, LoRA	Tokenized coordinates	RMSD, trajectory novelty
MLLM-DMD (Li et al., 17 May 2025)	ODEs, coordinate maps	MLLM zero-shot	Symbolic regressors	$R^2$, equation length, fitness

A plausible implication is that LLM-DMD frameworks eliminate traditional rigidity in model structure and basis selection, automating the search for coordinates, variables, and governing equations across domains. The demonstrated quantitative improvements and empirical adaptability suggest strong utility in scientific discovery, simulation, and control, though scalability, interpretability, and physical grounding remain active research areas.