Ergodic LQ Closed-Loop Optimal Control

• Ergodic LQ optimal control is a framework that minimizes long-run quadratic costs for linear stochastic systems using steady-state feedback laws.
• It employs algebraic Riccati equations and invariant measure analysis to ensure mean-square stability and admissibility under random or periodic coefficients.
• Extensions cover mean-field, risk-sensitive, and adaptive control scenarios, with practical applications demonstrated in aerospace and high-dimensional models.

Ergodic linear-quadratic (LQ) closed-loop optimal control addresses the synthesis and analysis of control laws that minimize long-run average (ergodic) quadratic performance criteria for linear stochastic systems, possibly with nontrivial mean-field (distribution-dependent) and time-varying or random coefficients. The ergodic regime focuses on steady-state or long-term average behavior, contrasting with discounted or finite-horizon models, and its mathematical underpinnings lie in stochastic control theory, Riccati equations, and invariant measure analysis. This field encompasses advances in theory for mean-field dynamics, systems with uncertainty, periodic/random coefficients, risk-sensitive objectives, as well as robustness under structural uncertainty.

1. Fundamental Formulation of the Ergodic LQ Closed-Loop Problem

At the core, the ergodic LQ closed-loop problem seeks an admissible control law that minimizes an ergodic cost functional associated with a controlled stochastic differential equation (SDE):

• State equation (general, possibly mean-field):

$$dx_t = [A\,x_t + \bar A\,m_t + B\,u_t + b]\,dt + [C\,x_t + \bar C\,m_t + D\,u_t + \sigma]\,dW_t,\quad m_t = \mathbb{E}[x_t]$$

with $x_t \in \mathbb{R}^n$, $u_t \in \mathbb{R}^m$, $W_t$ a standard Brownian motion.

• Quadratic cost functional (possibly mean-field):

$$J_{\rm erg}(u) = \limsup_{T\to\infty} \frac1{T} \mathbb{E} \int_0^T \bigl[ x_t^\top Q x_t + u_t^\top R u_t + 2x_t^\top S u_t + 2q^\top x_t + 2r^\top u_t + m_t^\top \bar Q m_t \bigr]\,dt$$

• Admissibility demands uniform moment bounds and the existence of a limiting ergodic law (e.g., a stationary distribution in the $W_2$ metric) (Bayraktar et al., 13 Feb 2025, Mei et al., 2020).

The ergodic framework is robust to transients: the cost is determined by the long-term, steady-state interplay between the process dynamics and feedback law, possibly involving invariant or periodic measures.

2. Algebraic Riccati Equations and Verification

The closed-loop optimal synthesis hinges on the solution of (generalized) algebraic Riccati equations (AREs) that arise from the stationary Hamilton–Jacobi–Bellman (HJB) equation:

• Quadratic value function ansatz yields coupled Riccati equations for the state and mean-field terms:

$$A^\top P + P A + C^\top P C + Q - (P B + C^\top P D + S^\top) (R + D^\top P D)^{-1} (B^\top P + D^\top P C + S) = 0$$

with analogous equations for mean-field (e.g., $\Pi$), affine terms ($P_1$, $p$, $p_1$), and ergodic value $c_0$ (Bayraktar et al., 13 Feb 2025, Mei et al., 2020, Wu et al., 8 May 2025).

• Random/periodic coefficients generalize the Riccati equation to backward stochastic Riccati equations (BSREs) or periodic Riccati ODEs. Existence and uniqueness in the random periodic setting are obtained via monotone BSDE iteration and $L^2$-contraction (Wu et al., 13 Jan 2026).
• Finiteness and admissibility may hold even with indefinite weights, under suitable Riccati inequalities. Existence of a stabilizing solution implies the ergodic problem is well-posed (Mei et al., 2020).

3. Structure of the Ergodic LQ Optimal Feedback

The optimal closed-loop control is linear (affine if nonhomogeneous) in the state and, for mean-field systems, includes a term dependent on the mean:

• General form (mean-field LQ):

$$u_t^* = \Theta^* (x_t - m_t) + \bar\Theta^* m_t + \theta^*$$

with explicit formulas for gains:

$$\Theta^* = - (R + D^\top P D)^{-1}(B^\top P + D^\top P C + S)$$

$$\bar\Theta^* = - (R + D^\top P D)^{-1}(B^\top \Pi + D^\top P \bar C + S)$$

and

$$\theta^* = - (R + D^\top P D)^{-1}(B^\top p + D^\top P \sigma + r)$$

(Bayraktar et al., 13 Feb 2025, Wu et al., 8 May 2025). In purely homogeneous cases ($C = \bar C = \sigma = b = q = r = 0$), the feedback specializes to classical LQ structure (Bayraktar et al., 13 Feb 2025, Mei et al., 2020).

• For random periodic coefficients, the optimal law becomes periodic in $t$:

$$u^*(t) = -R(t)^{-1}[B(t)^\top P(t) + D(t)^\top P(t)C(t)]X(t) - R(t)^{-1}[B(t)^\top \eta(t) + \rho(t)]$$

where $P(t), \eta(t)$ are solutions to periodic BSREs and BSDEs (Wu et al., 13 Jan 2026).

• Stationarity and Invariant Measures: When the closed-loop is mean-square stabilizing, an invariant Gaussian measure exists, with the cost evaluated via this measure (Mei et al., 2020, Wu et al., 8 May 2025).

4. Asymptotic Analysis: Equivalence, Turnpike, and Invariant Measures

A distinctive trait of ergodic LQ control is the equivalence between the infinite time-averaged cost and the expectation with respect to an invariant or periodic measure:

• Reduction to one-period average: For periodic (or random periodic) coefficients, the ergodic cost over $[0,\infty)$ reduces to an expectation over a single period, weighted by the invariant (periodic) measure (Wu et al., 13 Jan 2026, Wu et al., 8 May 2025).
• Turnpike property: The finite-horizon LQ optimal state-control pair converges exponentially fast to the ergodic (infinite-horizon) solution, except near initial and terminal boundary layers. Quantitatively, for $t$ away from boundaries,

$$\mathbb{E}[|X_T(t)-\bar X(t)|^2 + |u_T(t)-\bar u(t)|^2] \leq K' (e^{-\lambda' t} + e^{-\lambda'(T-t)})$$

This demonstrates transient region localization and long-run stability (Bayraktar et al., 13 Feb 2025).

• Convergence of Riccati trajectories: Time-varying Riccati equations for the finite-horizon problem converge exponentially to their algebraic (ergodic) limits (Bayraktar et al., 13 Feb 2025).
• Invariant and periodic measures for state laws: Under stabilizing feedback, Markov transition kernels contract in Wasserstein distance to unique invariant or periodic measures, capturing steady-state behavior (Wu et al., 8 May 2025).

5. Extensions: Mean-Field, Periodic, Random, and Adaptive Ergodic LQ Control

Recent work generalizes classical ergodic LQ theory in several directions.

• Mean-field systems: Coupling through the empirical mean of the state introduces additional Riccati equations, periodic measure structures, and requires completion of the square in both state and mean (Bayraktar et al., 13 Feb 2025, Wu et al., 8 May 2025).
• Random periodic and stochastic coefficients: Ergodic theory for random periodic stochastic systems relies on random periodic mean-square exponential stability. Backward stochastic Riccati equations with random periodicity yield time-varying, pathwise-periodic feedback gains and cost representations over a single period (Wu et al., 13 Jan 2026).
• Indefinite weights: Ergodic LQ optimal control remains well-posed under indefinite $Q$ and $R$ if suitable Riccati inequalities admit solutions. Regularization with an $\varepsilon$-penalized cost (for small $\varepsilon > 0$) yields existence and computes the minimal ergodic value as $\varepsilon \to 0$ (Mei et al., 2020).
• Adaptive LQ control: In the presence of unknown system matrices, certainty-equivalence adaptive control with appropriate weighted least-squares estimation, random regularization, and diminishing excitation achieves ergodic optimality. The key is that parameter estimation in the cost-relevant subspace is consistent, and the diminishing exploration noise has vanishing contribution to the ergodic cost (Liu et al., 2024).

6. Risk Constraints, Robustness, and Numerical Performance

Recent developments address the robustification of ergodic LQ policies to unpredictable or heavy-tailed noise environments:

• Ergodic-risk criteria quantify long-term variance or tail risk by considering limits of martingale increments associated with risk functionals. Asymptotic variance of empirical costs admits closed-form expressions given the invariant distribution (Talebi et al., 10 Feb 2025).
• Constrained LQR synthesis incorporates a bound on the asymptotic risk, formulating a constrained optimization over the space of stabilizing linear feedbacks, with solutions characterized via modified discrete algebraic Riccati equations and resolved through primal-dual algorithms (Talebi et al., 10 Feb 2025).
• Stability and functional CLT: Exponential mean-square stability of the closed-loop system ensures unique invariant measures, bounded second moments, and enables functional central limit theorems (for long-run empirical costs) (Talebi et al., 10 Feb 2025).
• Numerical examples on high-dimensional aerospace models demonstrate that optimal ergodic risk-constrained controllers suppress large noise-induced state excursions at minimal increase in average cost (Talebi et al., 10 Feb 2025).

---

References:

• (Wu et al., 13 Jan 2026): Ergodic LQ control with random periodic coefficients
• (Wu et al., 8 May 2025): Ergodic LQ mean-field control with periodic coefficients
• (Bayraktar et al., 13 Feb 2025): Ergodicity and turnpike for mean-field LQ control
• (Liu et al., 2024): Adaptive ergodic LQ control
• (Talebi et al., 10 Feb 2025): Ergodic-risk constrained LQR
• (Mei et al., 2020): Classical ergodic LQ with indefinite weights

All detailed formulas, proof structures, and theoretical advances above can be found in the cited references. The ergodic LQ closed-loop control literature continues to expand, addressing increasingly complex dynamics, information structures, and performance objectives.