Deep Learning for Dynamic Programming with Recursive Utility

Published 5 Jul 2026 in q-fin.CP, cs.LG, econ.EM, math.OC, and stat.ML | (2607.04278v1)

Abstract: We propose the first deep learning algorithm, the Certainty Equivalent Learning (CEL) algorithm, for solving high-dimensional discrete-time dynamic programming problems with recursive utility. Dynamic programming with recursive utility is numerically challenging because the recursive utility does not have an explicit representation and the Bellman equation contains a certainty equivalent that is difficult to evaluate. The CEL algorithm learns this certainty-equivalent value directly with neural networks and jointly approximates value functions, policy functions, and certainty-equivalent functions. The CEL algorithm is mesh-free and simulation-based, allowing high-dimensional state and control spaces, and does not rely on Euler equations, first-order conditions, or differentiability of the state transition function. The CEL algorithm also works for dynamic programming problems with expected utility as expected utility is a special case of recursive utility. We apply the CEL to discounted linear exponential quadratic Gaussian control, small-noise robust control, Epstein-Zin DSGE, and multivariate strategic asset allocation problems. Compared with closed-form and VFI-based benchmarks, the CEL delivers accurate value and policy approximations, remains effective in high-dimensional problems, achieves accuracy comparable to VFI in the small-noise robust-control case, and produces out-of-sample Bellman errors and Euler or first-order residuals that are in the range from 1.0e-4 to 1.0e-3 for most problems.

Abstract PDF Upgrade to Chat

Authors (2)

Summary

The paper introduces the CEL algorithm, a mesh-free neural approach to solve high-dimensional DP problems with recursive utility.
It employs modular neural architectures, including multi-network designs, to effectively manage nonlinear certainty-equivalent operators.
Empirical results demonstrate high numerical accuracy and robust policy evaluation across a variety of challenging economic models.

Deep Learning for High-Dimensional Dynamic Programming with Recursive Utility: The Certainty Equivalent Learning Algorithm

Overview and Motivation

Solving discrete-time dynamic programming (DP) problems with recursive utility is fundamental in macroeconomics and finance, with applications extending to stochastic growth, robust control, and asset allocation. Classical grid-based or projection-based numerical methods have limited scalability due to the exponential growth of computational costs with state and control dimension. Furthermore, recursive utility introduces formidable computational challenges: the Bellman equation incorporates a certainty-equivalent operator—a nonlinear, state-control-dependent functional of the conditional distribution of future value—which does not admit a closed-form in most settings. The Certainty Equivalent Learning (CEL) algorithm proposed in this work (2607.04278) provides a mesh-free, neural network-based, simulation-driven framework that directly circumvents these obstacles, enabling accurate, scalable solution of high-dimensional DP problems with general recursive utility specifications.

Recursive Utility and Computational Challenges

Recursive preferences, typified by the Epstein–Zin or risk-sensitive models, separate risk aversion from intertemporal substitution, enabling modeling of agent attitudes to long-run risk and ambiguity. The associated Bellman equations embed a certainty-equivalent functional $V_c(s, c) = f^{-1}\left(\mathbb{E}[f(V(s')) | s, c]\right),$ where transitions $s'=\psi(s, c, z')$ may be nonlinear or non-Gaussian, and $f$ typically encodes risk-sensitive or Epstein–Zin transformations.

Several nontrivial computational challenges arise:

Lack of explicit value representation: The value function must be found as a fixed point, often without an explicit expansion in future utilities.
Nonlinear certainty-equivalent term: The certainty equivalent is a high-dimensional, nonlinear expectation over future value; classical sampling yields biased estimates due to Jensen's inequality.
State-control dependence: Policy improvement requires evaluating certainty equivalents at off-policy candidate controls.
Bias in naive MC estimation: For nonlinear $f$ , plug-in estimators are generically biased, requiring dedicated function approximation.
Optimization instability: Nonconvex neural training, coupled equations (policy, value, certainty equivalent), and recursive bootstrapping amplify errors across updates.

These features make recursive utility DP problems significantly harder to solve than their expected utility counterparts.

The Certainty Equivalent Learning (CEL) Algorithm

CEL provides a neural approximation framework custom-tailored to the requirements of recursive utility. The approach is characterized by several architectural and algorithmic design features:

Neural Architecture Variants

Three-network architecture: Separates value function ( $V$ ), policy function ( $c$ ), and certainty-equivalent function ( $V_c$ ) into independent neural modules.
Four-network architecture: Further decomposes the certainty-equivalent module into an expected conditional value network ( $V_e$ ) and a nonlinear Jensen gap network ( $D$ ): $V_c = V_e - D$ .
Two-network architecture: Removes the value network, representing the current value implicitly as a function of learned policy and certainty-equivalent networks.

This modularity enables both flexibility and targeting, critical for handling high curvature or high-dimensional nonlinearities in the certainty-equivalent transformation.

Learning and Optimization Protocol

The primary innovation is the direct learning of the certainty-equivalent function as a separate object, indexed by both state and control, rather than repeated noisy estimation via conditional Monte Carlo. The optimization proceeds in alternating phases:

Certainty-equivalent update: Fit neural $s'=\psi(s, c, z')$ 0 to match simulation-based samples of the desired certainty-equivalent target; in four-network variants, fit $s'=\psi(s, c, z')$ 1 and $s'=\psi(s, c, z')$ 2 separately.
Policy update: Maximize the Bellman operator, treating the learned $s'=\psi(s, c, z')$ 3 as fixed.
Value function update: (When present) Fit the value network to the Bellman target built from the current policy and certainty-equivalent outputs.

Algorithmic stabilization employs:

Mesh-free, pathwise simulation: State transitions are generated by simulating the model, not by interpolation or grid construction.
Exploratory control perturbation: Action space exploration is facilitated by additive noise (Gaussian or Sobol sequence) during simulation to ensure coverage of the relevant state-control manifold.
Target networks: Slowly updated copies of key networks provide stabilization during the bootstrapped learning of bellman targets, suppressing instability from rapid parameter updates.
Delayed policy update: Policies are only updated every few iterations to mitigate error propagation from value/bootstrap estimation.

CEL is mesh-free, does not require differentiability or tractable densities in the transition function, and is broadly applicable to generic discrete-time Markovian models.

Numerical Results and Empirical Validation

CEL is benchmarked across a spectrum of problems, illustrating both scalability and accuracy:

Linear-exponential-quadratic Gaussian (LEQG) control: In both moderate ( $s'=\psi(s, c, z')$ 4, $s'=\psi(s, c, z')$ 5) and high-dimensional ( $s'=\psi(s, c, z')$ 6, $s'=\psi(s, c, z')$ 7) regimes, CEL's value and policy networks agree with analytical solutions to within $s'=\psi(s, c, z')$ 8. Certainty-equivalent function learning avoids error amplification seen in naive MC plug-in methods.
Robust (small-noise) control with non-explicit solutions: CEL closely matches value and policy functions produced by high-accuracy value function iteration (VFI). Where analytical solutions are available (homothetic log utility), CEL exhibits lower or comparable absolute errors ( $s'=\psi(s, c, z')$ 9– $f$ 0) relative to grid-based VFI, with reduced local oscillation in policy outputs.
DSGE with Epstein–Zin utility and stochastic volatility: The method reliably solves a 3D state, 2D-control recursive macro model, achieving relative Bellman and Euler residuals $f$ 1– $f$ 2.
Strategic asset allocation: In a portfolio model with multivariate VAR state and multiple risky assets, CEL delivers stable value and policy mappings, with Bellman and static condition errors below $f$ 3 across the state manifold.

Notably, residual-based out-of-sample diagnostics (Bellman errors, Euler/FOC residuals) consistently confirm the internal consistency and numerical accuracy of the learned solutions, across both moderate and high-dimensional settings.

Theoretical and Practical Implications

CEL demonstrates several significant implications for computational economics and quantitative finance:

Scalability: Mesh-free simulation and direct certainty-equivalent learning enable tractable solution of problems far beyond the practical limits of grid-based DP—even with hundreds of state/control variables.
Generalizability: The approach does not rely on expected utility's linearity, differentiability, or invertibility assumptions—recursive utility, robustness, and state-dependent nonlinearities are handled natively.
Numerical accuracy: Residual-based diagnostics and direct comparison to closed-form benchmarks demonstrate that neural methods, when carefully designed, do not sacrifice internal consistency even in complex, high-dimensional models.
Robustness: Algorithmic stabilization (target networks, policy delays, explicit certainty-equivalent estimation) prevents bootstrap instability typical in naïve deep value iteration with nonlinear preference operators.

Given these properties, CEL is poised to alter the practical feasibility of quantitative solution methods for dynamic macro-financial models, robust control, heterogeneous agent models, and decision-theoretic contexts featuring recursive risk.

Future Directions

The research suggests multiple promising extensions:

Heterogeneous-agent economies: High-dimensional models with idiosyncratic shock structures or market frictions.
General equilibrium and rational expectations: Neural modules for expectations formation, learning-based market clearing, or recursive competitive equilibrium.
Model uncertainty and ambiguity: Integration with nonparametric ambiguity sets or data-driven robust control.
Continuous-time extensions: Adaptation to BSDE/HJB-based recursive problems, potentially leveraging deep FBSDE/PDE solvers for further tractability.

Combining CEL with recent advances in self-play, meta-RL, uncertainty quantification, or online learning could further expand its domains of applicability.

Conclusion

Certainty Equivalent Learning provides a rigorous, scalable algorithmic foundation for solving high-dimensional recursive utility dynamic programming problems. By targeting the core challenge—nonlinear certainty-equivalent operators—through explicit state-control function learning and robust deep RL stabilization techniques, CEL both generalizes and surpasses prior deep methods developed for expected-utility DP. Numerical evidence demonstrates that the approach delivers solutions which are both numerically accurate and tractable for environments previously regarded as computationally infeasible. CEL thus meaningfully broadens the frontier for analysis of dynamic stochastic models in applied economics and finance (2607.04278).

Markdown Report Issue