Thermodynamics of learning physical phenomena

Abstract: Thermodynamics could be seen as an expression of physics at a high epistemic level. As such, its potential as an inductive bias to help machine learning procedures attain accurate and credible predictions has been recently realized in many fields. We review how thermodynamics provides helpful insights in the learning process. At the same time, we study the influence of aspects such as the scale at which a given phenomenon is to be described, the choice of relevant variables for this description or the different techniques available for the learning process.

• The paper introduces thermodynamics-based neural architectures, such as TANNs, that integrate conservation laws and free energy potentials to improve simulation accuracy.
• It employs Physics-informed Neural Networks and Hamiltonian Neural Networks to embed physical constraints, reducing overfitting and enhancing reliability.
• The research outlines a multiscale approach leveraging statistical mechanics and dynamical systems theory to accurately model both conservative and dissipative processes.

Thermodynamics of Learning Physical Phenomena

Introduction

The paper "Thermodynamics of learning physical phenomena" (2207.12749) explores the interfacing of machine learning with the principles of thermodynamics to enhance the understanding and prediction of physical phenomena. Thermodynamics, with its macroscopic scope, offers a paradigmatic inductive bias that assists in guiding machine learning approaches towards more accurate and thermodynamically consistent predictions.

The Paradigms of Scientific Discovery

Historical progression in science is marked by paradigms—empirical, theoretical, and simulation-based—as noted in the 2009 book that proposed the fourth paradigm of data-intensive scientific discovery. This paper discusses the burgeoning fifth paradigm, where simulations generate data, contributing to the discovery of phenomena like dark matter and protein folding. The importance of correctly selecting scales and variables informed by thermodynamic principles is crucial, as illustrated by early conceptual errors such as Galileo's in beam theory.

Machine Learning in Simulating Physical Phenomena

Machine learning, particularly in developing digital twins and learned simulators, significantly boosts simulation accuracy and efficiency. These models are faster yet maintain accuracy, free from dependency on traditional simulation models. However, issues like overfitting and data perturbations constrain their widespread acceptance. The paper emphasizes the integration of physics-driven inductive biases, such as those offered by Physics-informed Neural Networks (PINNs) and the more comprehensive framework of Thermodynamics-based Artificial Neural Networks (TANNs).

Statistical Mechanics: Choosing the Right Scale

Learning physical laws can range from molecular dynamics, governed by Hamiltonian mechanics, to coarse-grained thermodynamic descriptions. Each level of description offers specific insights and constraints, governed by inherently stochastic processes. The Fokker-Planck and Smoluchowsky equations capture these dynamics across scales, with thermodynamics representing invariant properties. This multiscale perspective is vital for ensuring that simulations encapsulate all significant physical interactions accurately.

Physics-informed Neural Networks (PINNs)

PINNs leverage known partial differential equations (PDEs) to enforce physical laws within neural network training, enabling solutions to complex physical systems by minimizing residuals of physical equations (see Equation in PINNs) alongside data fitting. Constraints such as symmetries and conservation laws can also be integrated into the learning architecture, enhancing prediction reliability and adherence to physical laws.

Thermodynamics-based Neural Architectures

Recent approaches like Thermodynamics-based Artificial Neural Networks (TANNs) and Variational Onsager Neural Networks (VONNs) explicitly incorporate thermodynamic laws into learning processes. By employing potentials such as the Helmholtz free energy and ensuring compliance with conservation and dissipation requirements, these approaches constrain neural network outputs to physically plausible spaces without overfitting against unseen data.

Dynamical Systems Perspective

Machine learning problems bear resemblance to dynamical systems, allowing the application of classical mechanics concepts such as Hamiltonian and Lagrangian dynamics to build more reliable models. Hamiltonian Neural Networks (HNNs) ensure conservation properties by aligning with symplectic integrative techniques, while Lagrangian frameworks leverage action minimization principles to reconstruct system dynamics.

Challenges in Dissipative Systems

Systems exhibiting dissipation require careful modeling beyond conservative frameworks. The metriplectic GENERIC (General Equation for the Non-Equilibrium Reversible-Irreversible Coupling) framework is well-suited to these systems, ensuring adherence to both conservation laws and non-negative entropy production. This formalism introduces a structured, thermodynamically consistent method for learning dynamics in open and dissipative systems.

Conclusion

The integration of thermodynamic principles within machine learning architectures results in more credible and constrained simulations, promoting trust when applied to industrial and scientific applications. These frameworks advance the field by reducing reliance on black-box models, consequently accelerating the adoption of machine learning solutions in real-world scenarios. Future developments will likely further strengthen this bond, offering instantaneous, parameter-sensitive simulation capabilities that adhere to physical laws.