New Designed Loss Functions to Solve Ordinary Differential Equations with Artificial Neural Network

Abstract: This paper investigates the use of artificial neural networks (ANNs) to solve differential equations (DEs) and the construction of the loss function which meets both differential equation and its initial/boundary condition of a certain DE. In section 2, the loss function is generalized to $n^\text{th}$ order ordinary differential equation(ODE). Other methods of construction are examined in Section 3 and applied to three different models to assess their effectiveness.

• The paper introduces restructured loss functions for ANN-based ODE solvers, achieving higher accuracy by effectively integrating differential equations with their constraints.
• The methodology is validated through experiments on models such as Newton's law of cooling and motor suspension, demonstrating significant loss reduction compared to traditional methods.
• The approach generalizes to higher-order ODEs and suggests promising future research directions, including extensions to complex PDE scenarios.

New Designed Loss Functions to Solve Ordinary Differential Equations with Artificial Neural Network

Introduction

The paper "New Designed Loss Functions to Solve Ordinary Differential Equations with Artificial Neural Network" (arXiv ID: (2301.00636)) explores the use of artificial neural networks (ANNs) as an innovative method for solving ordinary differential equations (ODEs). Traditional techniques for solving differential equations primarily rely on domain discretization methods such as finite-difference, finite-volume, and boundary-element methods. These approaches, however, face limitations in handling high-dimensional equations or irregular domains, and often require interpolation for approximations outside mesh points.

ANNs offer a promising alternative by functioning as universal function approximators capable of producing continuous and differentiable solutions across entire domains without intricate meshing. The primary challenge in employing ANNs for ODEs is integrating initial and boundary conditions into a single network framework. This paper addresses this challenge by proposing reconstructed loss functions that adhere to both the differential equation and its constraints.

The disadvantage of the existing Differential Geometric Multiresolution (DGM) algorithm lies in its simplistic combination of separate loss functions for the main equation, initial conditions, and boundary conditions, which tends to compromise the accuracy of solutions. This paper extends the work of Lagaris [7] by refining the method to construct loss functions that achieve precise reconciliation of differential equations and constraint conditions.

Figure 1: DGM algorithm for solving PDE

Construction of Loss Functions

The paper establishes a general formula for reconstructed loss functions applicable to $n^{\text{th}$ order ODEs. Initially, it focuses on defining a general differential equation:

$$F(x, u, Du, D^{2}u,...,D^{m}u) = 0, \quad x \in \Omega \subset \mathbb{R}^{n}$$

Here, $u$ represents the solution, and the derivatives $D^{m}u$ are structured as partial derivatives of varying orders. The ANN inputs are the discretized domain points $\Omega$, resulting in a system of equations. The paper details reconstruction techniques for first-order and second-order ODEs, focusing on transforming these into unconstrained problems by modifying the solution function as follows:

$$u_{NN}(\mathbf{x}, \mathbf{p}) = A(\mathbf{x}) + G(\mathbf{x}, N(\mathbf{x}, \mathbf{p}))$$

This transformation leverages a neural network output, accounting for constraints and the ODE itself, to minimize singular losses effectively.

The paper provides detailed examples for first-order and second-order ODEs, showcasing specific loss function constructions under various boundary conditions. It then generalizes these methods to higher-order ODEs, ensuring adaptability of the approach for complex systems.

Application in Models

Newton's Law of Cooling

This model addresses the temperature change rate of a body relative to its environment. It is described by the differential equation:

$\frac{dT}{dt} = r (T_{env} - T)$

With initial parameters set as $T(0) = 100^\circ C$, $T_{env} = 10^\circ C$, and $r = 0.5$, the analytical solution is derived as $T(t) = 10 + 90e^{-0.5t}$. Upon implementation in an ANN framework using 7 different reconstruction functions, the exponential reconstruction emerged as superior, achieving the smallest loss.

Figure 2: Comparing losses of 7 constructed functions in model of Newton's law of cooling

Motor Suspension System

This model reflects the dynamics of a motor's suspension using a mass-spring-damper system modeled by a second-order ODE. Parameters include the mass $m$, spring constant $k$, and damping coefficient $c$. Solving this yields $x(t) = 3.5e^{-8t}-\frac{19}{6}e^{-12t}$. Performance evaluation identified logarithmic and polynomial reconstructions as most effective.

Figure 3: Comparing losses of 7 constructed functions in model of motor suspension system

Home Heating

Utilizing a system of first-order ODEs, home heating models simulate temperature variations across different areas, subject to external conditions and heating influence. The performance of diverse reconstruction functions was assessed, with exponential and hyperbolic forms showing promising results.

Figure 4: Comparing losses of 7 constructed functions in home heating model

Conclusion and Future Research

The paper successfully illustrates the effectiveness of ANNs in solving ODEs with a restructured approach to loss functions, yielding precise solutions across various models. Future research could explore extending the current approach to include alternative reconstruction forms for higher-order ODEs. Moreover, tailoring general construction formulas for partial differential equations (PDEs) could amplify the applicability of this method across fields demanding complex modeling capabilities.