Hamilton-Jacobi-Bellman Optimality

• Hamilton-Jacobi-Bellman optimality conditions are nonlinear PDEs that characterize the value function by minimizing a Hamiltonian built from system dynamics and cost functions.
• They extend classical control frameworks by incorporating entropy regularization and weak solution concepts, such as viscosity and proximal solutions, for broader applicability.
• Modern advancements leverage grid-free solvers and learning-based methods to overcome computational challenges and ensure convergence to stable, optimal feedback laws.

The Hamilton-Jacobi-Bellman Optimality Conditions

The Hamilton-Jacobi-Bellman (HJB) optimality conditions constitute the central necessary and, under regularity, sufficient criteria for optimality in deterministic and stochastic continuous-time optimal control problems governed by dynamic programming. These conditions take the form of nonlinear partial differential equations (PDEs), characterizing the value function—namely, the infimum of cost-to-go functionals over all admissible control processes—through the minimization of a Hamiltonian constructed from the system dynamics and running cost. Modern developments extend the HJB framework to generalized control spaces (e.g., relaxed or randomized strategies), include entropy or KL-divergence regularization, and admit weak (e.g., viscosity, Sobolev, or proximal) solution concepts.

1. Classical HJB Optimality Conditions in Deterministic Continuous Time

For a deterministic control system $\dot x=f(x,u)$, $x(t)\in\mathbb{R}^n$, $u(t)\in U\subseteq\mathbb{R}^m$, and running cost $r(x,u)$ with terminal cost $q(x)$, the value function

$$V(t,x) = \inf_{u(\cdot)} \left\{ \int_t^T r(x(s),u(s))\,ds + q(x(T)) \right\}$$

must satisfy the Bellman optimality principle. The formal infinitesimal principle yields the HJB PDE: $$\partial_t V(t,x) + \min_{u\in U}\left\{\, r(x,u) + \nabla_x V \cdot f(x,u) \, \right\} = 0,\quad V(T,x)=q(x)$$ The minimization structure of the Hamiltonian encapsulates both system dynamics and cost, defining

$$H(x,p) = \min_{u\in U}\left\{ r(x,u)+p\cdot f(x,u) \right\}$$

The value function $V$ is required to be a viscosity solution since classical (smooth) solutions seldom exist outside linear-quadratic settings. For continuous and convex Hamiltonians, viscosity theory guarantees existence and uniqueness under mild assumptions (Kim et al., 2020).

2. Entropy-Regularized HJB and Maximum Entropy Control

Maximum entropy control generalizes the classical HJB theory by regularizing the cost functional with a control distribution entropy penalty: $$J^\alpha_{x_0}(\mu) = \int_0^T \left[ \int_U r(x(t),u)\,\mu(t;du) - \alpha H(\mu(t;\cdot)) \right] dt + q(x(T))$$ where $$H(\mu)=-\int_U \frac{d\mu}{du}\log\frac{d\mu}{du} du$$ and $\alpha>0$ is the temperature. The associated value function $V_\alpha$ obeys a "soft" HJB equation: $$\partial_t V_\alpha(t,x) - H_\alpha(x, \nabla_x V_\alpha ) = 0$$ with the soft Hamiltonian (by Legendre transform and Boltzmann law)

$$H_\alpha(x,p) = \alpha \log \int_{U} \exp\left(-\frac{p\cdot f(x,u) + r(x,u)}{\alpha}\right) du$$

For control-affine systems ($f(x,u)=f_1(x)+f_2(x)u$, $r(x,u)=r_1(x)+\frac{1}{2}u^\top Ru$), the optimal $g^*(u)$ is a Gaussian, and the mean-field control law in the limit $\alpha\to 0$ recovers the classical deterministic feedback (Kim et al., 2020).

In the linear-quadratic setting, the value function ansatz yields an algebraic Riccati equation,

$$\lambda P +A^\top P +P A -P\,B\,R^{-1}B^\top P +Q=0$$

and the optimal policy is affine in $x$, confirming exact solvability in this regime.

The soft HJB admits efficient "grid-free" numerical solvers via generalized Hopf-Lax formulas and characteristic ODEs, enabling tractable computation even in high dimensions.

3. Viscosity, Proximal, and Weak Solution Frameworks

Classical solutions to the HJB equation are rare; instead, weak solution concepts have been developed:

• Viscosity solutions: A function $V$ is a viscosity solution if at any local maximum (minimum) against a smooth test function $\phi$, the subsolution (supersolution) inequality with the Hamiltonian holds. Viscosity theory, utilizing comparison principles for convex and Lipschitz Hamiltonians, guarantees uniqueness and stability of $V$ (Kim et al., 2020, Kim et al., 2019).
• Proximal solutions: In infinite-dimensional (e.g., uncertain or distributed parameter) settings, optimality conditions are formulated using the proximal subdifferential of the value function in Hilbert spaces, leading to characterizations via invariance of the value function's epigraph under feasible dynamics (Aronna et al., 2024).
• Sobolev (weak) solutions: For stochastic HJB equations with jumps or rough coefficients, one seeks adapted $L^2$-valued solutions in Gelfand triples, often reformulating the HJB as a backward stochastic evolution equation with jumps (BSEEJ) (Meng et al., 2020). These frameworks extend the HJB principle to general nonlinear, stochastic, and jump-diffusion systems.

4. Uniqueness, Stability, and Failure Modes

In finite (tabular) state settings, the Bellman operator is contractive and the HJB equation admits a unique solution. In continuous state spaces, especially for linear-quadratic systems, the HJB equation may have an exponentially large set of smooth solutions (e.g., $\binom{2n}{n}$ for LQR of state-dimension $n$), of which only one yields an admissible, stabilizing feedback law—characterized by positive definiteness (Lyapunov condition) and asymptotic stability (Hurwitz closed-loop) (You et al., 4 Mar 2025). Value-based learning without appropriate architectural constraints may converge to inadmissible, unstable solutions due to the combinatorial imbalance. Enforcing positive definiteness by construction in value-function parameterizations eliminates spurious solutions and ensures convergence to the stable optimum.

5. Stochastic HJB, Jump Processes, and Anticipative Control

For stochastic control problems, the dynamic programming principle yields backward stochastic partial differential equations for the value process $V$:

• Diffusions with jumps: The value function $(V,\Phi,\Psi)$ solves

$$-dV(t,x) = \inf_{u} \left\{ H(\cdots) + \int_E [\mathcal{I} V - g \cdot D V ] \nu(de) + \int_E \mathcal{I} \Psi \nu(de) \right\} dt - \Phi\,dW - \int_E \Psi\,\tilde\mu(de,dt)$$

subject to terminal conditions (Meng et al., 2020). Existence and uniqueness follow under super-parabolicity and Gelfand triple assumptions.

• Anticipative settings: When the controller has access to a dynamically enlarged filtration (anticipative scenario), the associated causal HJB involves Dupire's functional derivatives and an additional transport constraint reflecting noise anticipation (Bank et al., 11 Jul 2025). The optimality condition is enforced in the causal path-dependent sense.

6. Numerical, Learning-Based, and Structural Aspects

Classical HJB-based optimality characterizations are computationally challenging due to curse of dimensionality and boundary-value structure (TPBVPs). Recent methods include:

• Grid-free solvers: Generalized Hopf-Lax and characteristic ODEs allow high-dimensional computation in soft HJB, avoiding curse of dimensionality for certain classes (Kim et al., 2020).
• Learning-based adjoint methods: Supervised and reinforcement learning architectures can efficiently approximate the adjoint variables or policy maps required to generate optimal trajectories without iterative shooting (You et al., 2019).
• Finite-dimensional approximation: Variational and max–min approximations of the HJB (and the related infinite-dimensional LPs) provide constructive recipes for nearly optimal control in periodic and measure-theoretic problems (Gaitsgory et al., 2013).
• Feedback synthesis: Once a suitable (classical or viscosity) solution to the HJB is obtained, feedback control is synthesized via minimization of the Hamiltonian at each state (Li et al., 18 Dec 2025).

7. Regularity, Verification, and Economic Applications

Rigorous verification of candidate solutions as true value functions requires sufficient regularity (differentiability/Malliavin-derivatives) to demonstrate tightness of the Bellman principle. In stochastic linear-convex problems, regularity properties are established via forward–backward stochastic differential equations and gradient-descent mappings in Hilbert spaces, guaranteeing unique classical solutions for uniformly convex costs (Li et al., 18 Dec 2025). In economic dynamics, it has been shown that the HJB equation is only necessary and sufficient for optimality under appropriate existence, concavity, and transversality conditions; in their absence, multiple nonadmissible HJB solutions may arise and the actual value function may not satisfy the HJB equation at all (Hosoya, 2021, Hosoya, 2022).

This synthesis unites dynamic programming, stochastic analysis, viscosity theory, and modern computational methods in articulating the full structure of HJB optimality conditions for continuous-time optimal control.