Fixed-Point Physics-Informed ResNets

• Fixed-Point Physics-Informed ResNets are deep learning architectures that integrate fixed-point iteration schemes to encode physical laws in residual networks.
• The methodology employs both stationary and non-stationary ResNet blocks that leverage physics-informed residuals, achieving high accuracy with MSE improvements and enhanced computational efficiency.
• Training challenges such as spurious fixed points are addressed using strategies like curriculum learning and adaptive sampling to refine the loss landscape and maintain physical fidelity.

Fixed-Point Physics-Informed Residual Networks describe an emergent research paradigm at the intersection of deep learning, dynamical systems, and computational physics. Rooted in the mathematical correspondence between fixed-point iteration schemes and residual neural networks (ResNets), these architectures utilize physics-informed constraints—typically arising as residuals of differential or integral equations—to guide both the learning process and the optimization of neural network-based solution operators. This approach simultaneously leverages prior knowledge about the structure of physical systems and the representational power of deep networks to solve forward and inverse problems in computational modeling, with distinctive implications for training dynamics, optimization landscapes, and empirical performance.

1. Mathematical Foundations: Fixed Points and Residual Networks

The foundational objects in Fixed-Point Physics-Informed Residual Networks are steady-state ("fixed-point") solutions of operator equations governing physical dynamical systems. An autonomous dynamical system of the form $u_t = F[u]$ admits special solutions $u^*(x)$ satisfying $F[u^*] = 0$. These zero-residual solutions simultaneously reflect physically meaningful equilibria and act as critical points in the loss landscapes of physics-informed neural networks (PINNs) that encode the governing equations through a residual term $$r(t,x;\theta) = u_{\theta,t}(t,x) - F[u_\theta(t,x)]$$ (Rohrhofer et al., 2022).

From a numerical perspective, fixed-point iteration methods for linear systems, such as those arising in discretized volume integral equations (VIEs), operate by updating the iterate according to

$\mathbf{x}_{k+1} = \mathbf{x}_k + F(\mathbf{x}_k),$

where $$F(\mathbf{x}_k) = (\mathbb{A}^a)^{-1}(\mathbf{b} - \mathbb{A} \mathbf{x}_k)$$ and $\mathbb{A}^a$ is an approximate preconditioner. This iterative scheme is structurally identical to a ResNet block, which updates hidden states via

$$\mathbf{x}_{k+1} = \mathbf{x}_k + \mathcal{F}(\mathbf{x}_k; \mathcal{W}_k),$$

with the correspondence $$F(\mathbf{x}_k) \leftrightarrow \mathcal{F}(\mathbf{x}_k; \mathcal{W}_k)$$ (Shan et al., 2021). The residual $$\mathbf{R}_k = \mathbf{b} - \mathbb{A} \mathbf{x}_k$$ thus encodes both the error of the discrete physics and the learning signal for the neural update.

2. Network Architectures: Stationary and Non-stationary Designs

The PhiSRL framework explicitly implements the fixed-point iterations as series of learned ResNet blocks, forming either stationary or non-stationary update sequences. The stationary architecture (SiPhiResNet) utilizes a single convolutional neural network $\Psi^{Si}(\cdot; \Theta)$ for all iterations, reapplying its weights each time. The iteration takes the form:

$$\mathbf{R}_k = \mathbf{b} - \mathbb{A} \mathbf{x}_k, \quad \Delta_k = \Psi^{Si}(\mathbf{R}_k \oplus \mathbf{x}_k; \Theta), \quad \mathbf{x}_{k+1} = \mathbf{x}_k + \Delta_k,$$

where $\oplus$ indicates channel-wise concatenation.

By contrast, the non-stationary architecture (NiPhiResNet) assigns a distinct convolutional kernel $\Psi^{Ni}(\cdot; \Theta_k)$ to each iteration, integrating additional depth and flexibility:

$$\Delta_k = \Psi^{Ni}(\mathbf{R}_k; \Theta_k), \quad \mathbf{x}_{k+1} = \mathbf{x}_k + \Delta_k,$$

with $\Psi^{Ni}$ a five-layer, fully convolutional network for each $k$ (Shan et al., 2021).

Both designs are supervised by the mean squared error between the iterated output $\mathbf{x}_L$ and a reference solution $\mathbf{x}^*$, often produced by a classical numerical solver such as the Method of Moments (MoM). This architecture-level embedding of the governing physics residual enables the network to emulate a fixed number of physical iterations, with learned updates tailored to the statistical properties and operator structure of the problem.

3. Optimization Landscape and Fixed-Point Induced Local Minima

A signature distinction of Fixed-Point Physics-Informed Residual Networks is the impact of physical fixed points on the optimization landscape. In PINNs, any fixed-point solution $u^*(x)$ results in a vanishing physics-informed loss $$L_\text{physics}(\theta) = \|u_{\theta,t} - F[u_\theta]\|^2$$, regardless of whether initial or boundary data are respected. Consequently, spurious but mathematically valid fixed points can serve as local or even global minima in the parameter space.

Expanding $F(u^* + \delta u) \approx J \delta u$, where $J$ is the Jacobian of $F$ at $u^*$, establishes that the loss $$L_f \approx \langle \delta u, J^T J \delta u \rangle$$, making $\delta u = 0$ a local minimum. Even when $J$ possesses directions associated with instability (negative real eigenvalues), the positivity of $J^T J$ ensures these are always non-negative in the squared-residual loss, preventing escape from fixed-point traps (Rohrhofer et al., 2022).

This structural feature is empirically manifest in training PINNs on systems with nontrivial equilibria: e.g., undamped pendulum dynamics yielding convergence to the upright rather than the swinging solution, or Navier–Stokes simulations stabilizing at steady vortices instead of Kármán vortex shedding. Loss landscape visualization reveals distinct basins for true and spurious fixed-point solutions, whose depth and prevalence depend on factors such as the time interval and proximity of initial conditions to fixed points.

4. Numerical Experiments, Performance, and Generalization

Comprehensive studies on 2D electromagnetic scattering problems using PhiSRL demonstrate the effectiveness and universality of the fixed-point residual paradigm. In modeling with SiPhiResNet (3 iterations) and NiPhiResNet (7 iterations), both lossless and lossy scatterers were accurately resolved, achieving mean squared errors (MSE) to the order of $10^{-4}$ and $10^{-7}$, respectively. For lossless samples, typical test mean absolute errors in the real/imaginary components were approximately 0.0132/0.0134 (SiPhiResNet) and $3.8\times 10^{-4}/4.2\times 10^{-4}$ (NiPhiResNet) (Shan et al., 2021).

Key experimental findings:

• Both architectures maintained high accuracy and robust generalization on unseen scatterer shapes and out-of-range frequencies, with MSE levels rising by less than a factor of two—showing strong physics-driven transferability.
• Monotonic or rapidly plateauing MSE decay was observed with iteration number, reproducing the convergence kernel of classical fixed-point solvers.
• In computational efficiency, NiPhiResNet achieved comparable accuracy to BiCGSTAB (7 iterations) but at 0.0005 s per inference compared to 0.008 s, indicating a 94% reduction in computation time.

These results highlight the dual advantages of embedding physics residuals as both optimization constraints and learning targets: fast convergence in a fixed number of learned steps, and order-of-magnitude improvements over conventional iterative solvers.

5. Training Pathologies and Mitigation Strategies

The presence of spurious fixed-point minima in the loss landscape results in recognizable training pathologies, notably convergence to undesired equilibria under certain initializations or extended simulation times. In both ODEs and PDEs, such as the Allen–Cahn equation or vortex-shedding benchmarks, PINNs can become "stuck" at steady states different from the true time-dependent solution, particularly when exposed to large temporal domains or ICs near equilibrium.

To alleviate these traps, several strategies are employed (Rohrhofer et al., 2022):

• Domain decomposition and curriculum learning: Segment the temporal or spatial domain into sub-intervals, progressively increasing the window of optimization to avoid premature convergence to fixed-point solutions.
• Adaptive collocation sampling: Dynamically increase collocation density in regions of pronounced temporal or spatial evolution, focusing optimization on non-equilibrium behaviors.
• Loss function reweighting and penalization: Augment the physics-informed loss with penalties on the proximity to fixed points or explicit enhancement of initial/boundary condition terms.
• Residual connection (hard IC enforcement): Architectures of the form $u(t,x;\theta) = u_0(x) + t N(t,x;\theta)$ enforce initial conditions by construction, preventing trivial fixed-point solutions.
• Physics-informed initialization and spectral bias: Pre-training with short windows or using architectures (e.g., SIREN) suited for oscillatory solutions can bias the model away from equilibria.
• Jacobian-based regularization: Penalizing small Jacobian singular values can reduce the attraction to equilibrium regions.

Multiple mitigations are often applied concurrently to reshape the effective optimization landscape, reducing the influence of fixed points and enabling convergence to physically relevant solutions.

6. Role in Broader Physics-Informed Learning

The fixed-point residual perspective has broader implications beyond VIEs and classical PINNs. Casting learned iterative updates in analogy with physical or numerical solvers enables network architectures to inherit convergence properties, physical constraints, and inductive biases that reflect the underlying equations. The explicit tie between ResNet blocks and fixed-point iteration both clarifies the mechanism of residual learning and establishes a template for future physics-informed network designs (Shan et al., 2021, Rohrhofer et al., 2022).

Ongoing research continues to explore optimal network parameterization, data-collocation strategies, and regularization techniques to systematically address pathologies unique to physics-constrained learning in systems dominated by nontrivial equilibria. This suggests the potential for further advancement in generalization, interpretability, and computational efficiency across a range of scientific machine learning tasks governed by operator equations.