Physics-Informed Policy Iteration (PINN-PI)

• Physics-Informed Policy Iteration (PINN-PI) is a mesh-free framework that integrates physics-informed neural networks with dynamic programming to solve nonlinear optimal and stochastic control PDEs.
• It alternates between neural network-based policy evaluation and improvement using automatic differentiation and KL-divergence loss, ensuring convergence with rigorous L2 error bounds.
• PINN-PI demonstrates scalability and efficiency for high-dimensional systems, extending to robust control, stochastic differential games, and PDE-constrained applications.

Physics-Informed Policy Iteration (PINN-PI) is a mesh-free algorithmic framework that leverages physics-informed neural networks (PINNs) to solve nonlinear optimal control, stochastic control, and differential game problems posed as partial differential equations (PDEs) of Hamilton–Jacobi–Bellman (HJB) or Hamilton–Jacobi–Isaacs (HJI) type. PINN-PI systematically alternates between neural network-based policy evaluation and improvement, integrating automatic differentiation with classical dynamic programming constructs. The approach provides rigorous $L^2$ error bounds, ensures convergence, and demonstrates scalability for high-dimensional problems that are intractable for grid-based solvers.

1. Mathematical Formulation and Policy Iteration Paradigm

PINN-PI is grounded in the controlled diffusion framework and entropy-regularized control. For $d$-dimensional stochastic systems governed by

$$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$

the entropy-regularized cost for a relaxed policy $\pi$ is

$$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$

where $\rho_0$ is a reference measure, and $\lambda$ controls the regularization strength (Kim et al., 3 Aug 2025). The optimal value $V(x)=\sup_\pi V^\pi(x)$ solves the nonlinear elliptic entropy-regularized HJB,

$\rho V(x) = \sup_{\pi\in\Pcal(U)} \left\{\tfrac{1}{2}\mathrm{tr}\left(\Sigma(x) D^2 V(x)\right) + \int_U [b(x,u)\cdot\nabla V(x)+r(x,u)]\pi(du) - \lambda\int_U \ln\pi(u)\,\pi(du)\right\}$

Classical policy iteration alternates:

• Policy evaluation: Solve the linear PDE for $V^\pi$ given fixed $d$0.
• Policy improvement: Derive the updated policy by softmax or minimization based on value gradients.

For general stochastic control or differential games, PINN-PI applies to HJB/HJI equations of the form

$d$1

with Hamiltonian $d$2 encoding supremum/infinum (or minimax) operators over controls/disturbances (Yang et al., 21 Jul 2025, Wang et al., 26 Aug 2025). In robust control, Zubov-type PDEs are handled using similar max Hamiltonian formulations.

2. PINN-Based Policy Evaluation and Neural Architecture

PINN-PI approximates both the value function $d$3 and the policy $d$4 (when needed) with neural networks. Key implementation aspects:

• Value networks $d$5 and policy networks $d$6 parameterized by $d$7, $d$8.
• Policy evaluation: Minimize the mean-squared PDE residual,

$d$9

with $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$0 encoding the local HJB/HJI residual (Kim et al., 3 Aug 2025, Meng et al., 2024).

• Policy improvement: Fit the network $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$1 to the analytic target (softmax or argmin) using KL-divergence loss or supervised anchor points (Kim et al., 3 Aug 2025, Wang et al., 26 Aug 2025).
• Collocation points $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$2 sampled iid in state and $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$3 in action space, supporting mesh-free approximation. Networks typically have 4-6 layers, 50-128 units per layer, with $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$4 or $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$5 activations (Kim et al., 3 Aug 2025, Mukherjee et al., 2023).

3. Mesh-Free Implementation and Training Procedures

• Mesh-free spatial sampling: $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$6 to $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$7 collocation points per iteration, batch size $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$8.
• Automatic differentiation computes gradients and Hessians required for PDE residuals.
• Optimization uses Adam with learning rates $$dX_t = \left[\int_U b(X_t,u)\,\pi(X_t,du)\right] dt + \sigma(X_t) dW_t$$9–$\pi$0.
• Alternation: Each PI cycle iteratively refines $\pi$1 and $\pi$2 until the $\pi$3 norm of value update $\pi$4 falls below tolerance (Kim et al., 3 Aug 2025, Kim et al., 3 Aug 2025).

For high-dimensional systems, direct mesh-based solvers exhibit exponential growth in complexity; PINN-PI maintains tractability across dimensions up to $\pi$5 (Meng et al., 2024, Yang et al., 21 Jul 2025).

4. Error Decomposition and Theoretical Guarantees

PINN-PI admits rigorous error bounds via a three-term decomposition:

$\pi$6

Assuming uniform ellipticity, drift bounds, and Lipschitz continuity of the policy update, the total error remains $\pi$7 (policy network error plus PINN residual) with exponential convergence in the PI index [(Kim et al., 3 Aug 2025), Thm. 4.1; (Kim et al., 3 Aug 2025); (Yang et al., 21 Jul 2025), Thm. 3.3]. Lipschitz bounds ensure that errors in value gradients propagate linearly to policy updates.

Variants such as ELM-PI apply linear least-squares when the system dimension is small, while PINN-PI (deep networks) generalizes to high-dimensional scenarios (Meng et al., 2024).

5. Extensions to Robust Control and Differential Games

PINN-PI extends to:

• Robust region of attraction (RROA), generalized Zubov PDE: Policy iteration alternates linear PDE solve under fixed disturbance and pointwise maximization to find optimal disturbance. Rollout-based anchor points stabilize training to prevent singular flat solutions (Wang et al., 26 Aug 2025).
• Stochastic differential games: Policy iteration solves HJI equations via minimax optimization at the policy improvement step. The value function is approximated mesh-free, and controls/disturbances are updated via pointwise saddle-point problems leveraging automatic differentiation (Yang et al., 21 Jul 2025). Equi-Lipschitz properties ensure convergence without convexity.

6. Numerical Benchmarks and Empirical Performance

PINN-PI has demonstrated empirical scalability, accuracy, and robustness:

• High-dimensional LQR ($\pi$8): PINN-PI achieves smooth convergence to near-optimal values, maintaining $\pi$9 errors below $$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$0. SAC (Soft Actor-Critic) requires orders of magnitude more data and plateaus for $$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$1 (Kim et al., 3 Aug 2025, Kim et al., 3 Aug 2025).
• Nonlinear benchmarks (pendulum, cartpole): PINN-PI stabilizes in fewer iterations, handles strong noise, and achieves higher rewards than model-free RL algorithms (Kim et al., 3 Aug 2025, Kim et al., 3 Aug 2025).
• Fluid-cooled battery packs (1D coupled PDE): Hybrid PINN-PI actor-critic framework attains fourfold improvement in sample efficiency compared to PPO, exploiting physical structure in the value network (Mukherjee et al., 2023).
• Robust region of attraction (2D Van der Pol, 10D decoupled): Rollout-enhanced PINN-PI achieves contour-accurate RROA estimates, avoiding singularities and scaling to $$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$2 (Wang et al., 26 Aug 2025).
• Differential games (publisher-subscriber, path planning): PINN-PI outperforms direct PINN solvers in accuracy and smoothness, maintaining relative $$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$3 errors $$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$4 in $$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$5 (Yang et al., 21 Jul 2025).

Sample performance metrics:

Benchmark	Dimensionality	PINN-PI $$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$6 error	Reference method	Data efficiency
LQR	$$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$7	$$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$8	SAC	$$V^\pi(x) = \mathbb{E}\left[\int_0^\infty e^{-\rho t} \left(\int_U r(X_t,u)\,\pi(X_t,du) - \lambda\,\mathrm{KL}\left(\pi(X_t,\cdot)\Vert\rho_0\right)\right) dt\right]$$9 lower
Cartpole	$\rho_0$0	Higher reward	SAC	Faster converg.
Zubov (decoupled)	$\rho_0$1	Accurate contours	FDM	Matched
Publisher-Subscr.	$\rho_0$2	$\rho_0$3	Direct PINN	Superior

PINN-PI maintains polynomial runtime scaling in practice, with wall-clock times comparable or superior to SAC and dramatically better than grid-based solvers as $\rho_0$4 increases (Kim et al., 3 Aug 2025, Meng et al., 2024).

7. Rigorous Verification, Stability, and Application Domains

Stability certification via formal verification (SMT solvers such as dReal) permits Lyapunov-region guarantees for the learned controllers, especially in nonlinear settings (Meng et al., 2024).

PINN-PI is applicable across domains:

• High-dimensional stochastic control (robotics, finance)
• Robust region estimation in control and safety analysis
• Stochastic differential games and multi-agent reinforcement learning
• PDE-constrained optimal control in energy, battery management, and path planning

A plausible implication is that direct embedding of physical PDE structure and dynamic programming into neural policy iteration yields scalable, verifiable control solutions robust to dimensionality and model regularization requirements.

• "Physics-informed approach for exploratory Hamilton--Jacobi--Bellman equations via policy iterations" (Kim et al., 3 Aug 2025)
• "Neural Policy Iteration for Stochastic Optimal Control: A Physics-Informed Approach" (Kim et al., 3 Aug 2025)
• "Solving nonconvex Hamilton--Jacobi--Isaacs equations with PINN-based policy iteration" (Yang et al., 21 Jul 2025)
• "Learning Robust Regions of Attraction Using Rollout-Enhanced Physics-Informed Neural Networks with Policy Iteration" (Wang et al., 26 Aug 2025)
• "Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification" (Meng et al., 2024)
• "Actor-Critic Methods using Physics-Informed Neural Networks: Control of a 1D PDE Model for Fluid-Cooled Battery Packs" (Mukherjee et al., 2023)