Deep Policy Iteration for High-Dimensional Mean-Field Games with Regenerative Reformulation

Abstract: This paper develops a deep policy iteration method for high-dimensional finite-horizon mean-field games. We reformulate the game as a regenerative problem with deterministic cycles, which allows policy evaluation (PE), policy improvement (PI), and population measure estimation to be carried out cycle by cycle. Within this formulation, we approximate the population measure by a particle system and update it using a one-step random mapping induced by the Euler-Maruyama discretization of the state dynamics. This update transports a mini-batch of particles from one cycle to the next, avoiding sequential trajectory simulation over the entire time horizon at each iteration. The PE and PI subproblems are formulated through the relation between consecutive cycles, with adversarial training used for evaluation and averaged optimization used for improvement. The resulting method is efficient and scalable in high dimensions, as it avoids the direct solution of the coupled Hamilton-Jacobi-Bellman and Fokker-Planck system, the full simulation of trajectories to estimate the population measure, the explicit computation of conditional expectations in policy evaluation, and pointwise optimization in policy improvement. Numerical experiments demonstrate that the proposed method effectively handles dimensions up to 10,000.

• The paper introduces a regenerative deep policy iteration method that reformulates finite-horizon mean-field games, overcoming traditional bottlenecks.
• It leverages neural networks and random particle ensembles to achieve scalable, mini-batch updates in dimensions up to 10,000.
• Empirical results demonstrate high accuracy, efficient convergence, and robustness across both linear-quadratic and nonlinear mean-field game scenarios.

Deep Policy Iteration for High-Dimensional Mean-Field Games with Regenerative Reformulation

Overview

The paper "Deep Policy Iteration for High-Dimensional Mean-Field Games with Regenerative Reformulation" (2604.26782) introduces a scalable deep reinforcement learning framework for the solution of high-dimensional, finite-horizon mean-field games (MFGs). The authors propose a regenerative reformulation that enables mini-batch updates and eliminates many of the bottlenecks associated with classical approaches, such as solving coupled Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck (FP) systems or computing high-dimensional integrals. This framework leverages neural networks for function approximation, random particle ensembles for state propagation, and adversarial Galerkin-type training. The approach efficiently handles dimensions up to $d = 10\,000$ in practice.

Regenerative Reformulation and Policy Iteration

Classical MFGs involve weak coupling among agents via costs and dynamics, and the equilibrium computation requires resolving a coupled HJB–FP system, suffering from the curse of dimensionality. The present approach reformulates the finite-horizon stochastic differential game as a regenerative process: the state process is composed of deterministic cycles of fixed length $T$ with a resetting mechanism, so the equilibrium problem becomes that of finding a stationary feedback control and state occupancy measure over cycles.

Within this regenerative structure, policy iteration is naturally adapted: policy evaluation (PE) and policy improvement (PI) steps cycle-wise update the value function and feedback control. The equilibrium occupation measure is empirically approximated with mini-batched particles, transported from cycle to cycle via a one-step random mapping using Euler-Maruyama discretization. This update mechanism avoids explicit trajectory simulation over the full horizon and obviates the need for directly solving the FP equation.

Weak-Form Deep Policy Iteration Framework

Value and policy functions are represented by neural networks $v_\theta$ and $u_\alpha$ parameterized respectively by $\theta$ and $\alpha$. The mini-batch of state samples $\{X_t^m\}$ approximates the equilibrium occupation measure; this set is updated via a random mapping $\Phi$ (one-step Euler-Maruyama) that uses current parameters and particle ensemble statistics.

The PE/PI subproblems are formulated in a weak Galerkin-type form. For PE, the value network is updated adversarially by solving a min-max problem, with a test-function network $\rho_\eta$ parameterized by $\eta$ serving as the adversary. This design replaces conditional expectation constraints by empirical expectations, improving sample efficiency and stability. For PI, the control network update minimizes the averaged empirical residual, replacing costly pointwise Hamiltonian minimization by stochastic optimization in parameter space. The overall architecture enables sharing of mini-batch sample transitions across all algorithmic components, offering substantial computational savings and parallelism.

Algorithmic Implementation

The main algorithm alternates the following steps:

1. Empirical Measure Update: Mini-batch subsets of particles are advanced using $T$0, constructing an updated empirical measure.
2. Adversarial Policy Evaluation (PE): The value function network is trained by minimizing the squared expectation of test functions applied to the PE residual, adversarially maximized over the test-function network.
3. Policy Improvement (PI): The control network is trained by minimizing the empirical expectation of the PI residual.
4. Shared Transitions: Updates for all networks leverage the same set of sample transitions, improving computational efficiency.

The method avoids explicit FP equation solution, full path simulations, PDE derivatives, and pointwise Hamiltonian minimizations—bypassing many scalability limitations of prior work.

Numerical Results

The framework is tested on several classes of MFGs including high-dimensional linear-quadratic (LQ) games, nonlinear target-tracking games, and MFGs with nonlinear drift, diffusion, and interaction terms. Key empirical findings include:

• For LQ MFGs with dimensionality $T$1 ranging from $T$2 up to $T$3, the method achieves final relative errors (RE) and relative costs (RC) in the range of $T$4 to $T$5, demonstrating stable convergence and high accuracy across a drastic increase in state-space dimension.
• The algorithm's computational runtime and memory usage exhibit moderate growth with problem dimensionality, e.g., $T$6 vs $T$7 shows a $T$82x runtime increase and $T$97.5x memory usage increase, with robust convergence and no loss of solution quality.

Figure 1: Numerical results showcasing the convergence of error and cost metrics for several LQ-MFG settings at $v_\theta$0.

• For $v_\theta$1 and $v_\theta$2, empirical cost and value function predictions track the analytic solution closely along principal subspaces of the state domain.

Figure 2: High-dimensional ($v_\theta$3) results for the value function, error decay, and cost decay in LQ-MFG—demonstrating scalable and accurate training with deep policy iteration.

• For nonlinear and non-quadratic MFGs (e.g., target tracking with strong nonlinear interaction, and games with cost barriers), learned agent trajectories qualitatively match the expected bulk behavior (e.g., efficiently tracking moving targets while maintaining agent separation, or collectively avoiding penalty regions).

Figure 3: Successful target-tracking behavior of agents in a nonlinear MFG—trajectories and spatial distribution snapshots highlight the method’s robustness beyond LQ benchmarks.

• GPU memory and wall-clock requirements remain practical for batch sizes up to $v_\theta$4 samples, leveraging mixed-precision and data-parallel implementation.

Practical and Theoretical Implications

This work demonstrates that high-dimensional MFGs—previously intractable with classical PDE-based or naive deep learning methods—can be efficiently and accurately solved using policy-iteration in a regenerative, weak-form setting with deep neural function approximators and scalable mini-batch sample propagation. The method is flexible with respect to game specification and supports arbitrary policy/value architectures. The empirical results suggest that for numerous classes of MFGs, the curse of dimensionality can be significantly mitigated, rendering dimensions up to $v_\theta$5 computationally feasible on modern multi-GPU hardware.

The weak-form adversarial training avoids both high-order derivatives and restrictive requirements on interaction or cost structure. Empirical results indicate strong stability with respect to initialization and hyperparameters, and robust convergence to Nash equilibrium even with poor initial measure guesses.

Future Directions

The paper identifies several open directions:

• Theoretical Analysis: Rigorous convergence guarantees and non-asymptotic error bounds for deep policy iteration and regenerative formulations in MFGs.
• Generality: Extensions to settings with more complicated mean-field coupling (e.g., coupling through both drift and cost, common noise), and to uncontrolled population measures.
• Adaptivity and Expressivity: Incorporation of adaptive sample selection and more expressive network architectures tailored to specific problem geometry or symmetries.

Conclusion

The regenerative weak-form deep policy iteration algorithm provides an effective, scalable, and flexible solution concept for high-dimensional mean-field games. Empirical results underscore its accuracy, computational tractability, and robustness. The elimination of full-trajectory simulation and direct HJB–FP coupling, together with the use of adversarial test-function learning, positions this method as a strong candidate for real-world large-population stochastic control problems across finance, engineering, and social sciences.