Entropy-Regularized MDPs

• Entropy-regularized Markov Decision Processes are a class of MDPs that add a Shannon entropy bonus at each decision step to promote robust exploration and smooth optimization landscapes.
• The framework leverages a softmax Bellman operator and convex duality, establishing strong theoretical guarantees including contraction properties and convergence of policy iteration.
• Practical implementations like Soft Actor-Critic and natural policy gradient methods benefit from these methods through improved stability, convergence rates, and effective transcript error bounds.

Entropy-regularized Markov Decision Processes (MDPs), often referred to as "soft" MDPs, augment the classical expected-sum-of-rewards objective with a policy entropy bonus at every decision epoch. This regularization, typically instantiated via the negative Shannon entropy, facilitates robust exploration, smooths the optimization landscape, and underpins many deep reinforcement learning methods. Entropy regularization is canonically integrated via a "softmax" Bellman operator, which induces stochastic, full-support optimal policies. The entropy-regularized framework is embedded within a broader class of regularized MDPs, connects deeply to convex duality and mirror descent, and is central to the modern theory and practice of reinforcement learning.

1. Mathematical Formulation and Bellman Operators

Let $(S, A, P, r, \gamma)$ be a discrete-time, infinite-horizon, discounted MDP. For a stationary stochastic policy $\pi(\cdot|s)$, the entropy-regularized objective is

$$\max_{\pi} \, \mathbb{E}_\pi\left[\sum_{t=0}^\infty \gamma^t \left(r(s_t, a_t) + \lambda H(\pi(\cdot|s_t))\right)\right],$$

where $H(\pi(\cdot|s)) = -\sum_{a} \pi(a|s) \log \pi(a|s)$ is the Shannon entropy and $\lambda>0$ is the regularization parameter (Li et al., 2019).

The regularized Bellman optimality equations are

$$\begin{aligned} Q^*_\lambda(s,a) &= r(s,a) + \gamma \, \mathbb{E}_{s'|s,a}[V^*_\lambda(s')], \ V^*_\lambda(s) &= \lambda \, \log \sum_{a} \exp(Q^*_\lambda(s,a)/\lambda). \end{aligned}$$

The unique optimal policy adopts the Boltzmann (softmax) form

$$\pi^*_\lambda(a|s) = \frac{\exp(Q^*_\lambda(s,a)/\lambda)}{\sum_{b}\exp(Q^*_\lambda(s,b)/\lambda)}.$$

The entropy-regularized Bellman operator is a $\gamma$-contraction in $\|\cdot\|_\infty$; thus, $V^*_\lambda$, $Q^*_\lambda$, and $\pi^*_\lambda$ are unique (Li et al., 2019, Geist et al., 2019, Mai et al., 2020).

2. Convex Duality, Mirror Descent, and Relaxed Policy Iteration

The regularized Bellman update can be interpreted as the convex conjugate (Fenchel dual) of the negative entropy: $$\lambda \log \sum_a \exp(Q_a/\lambda) = \sup_{p \in \Delta_A} [\langle Q, p\rangle + \lambda H(p)],$$ with $H(p)$ strongly concave, and the unique maximizer is the softmax policy (Li et al., 2019, Geist et al., 2019).

This duality underpins a link to mirror descent and regularized (proximal) policy iteration (Neu et al., 2017, Geist et al., 2019). Modern algorithms such as Trust Region Policy Optimization (TRPO), Soft Actor-Critic (SAC), and Dynamic Policy Programming are instantiations of mirror descent or dual-averaging on this regularized objective. The convex-optimization viewpoint yields convergence guarantees, regret bounds, and error propagation rates for both exact and approximate dynamic programming schemes.

3. Theoretical Guarantees and Performance Error Bounds

Key properties and bounds include:

• Contraction/Uniqueness: The soft Bellman operator is a $\gamma$-contraction; the entropy-regularized MDP admits unique fixed points for value, state-action value, and policy (Li et al., 2019, Geist et al., 2019, Neu et al., 2017).
• Performance Error: The regularized value function $V^*_\lambda$ is within

$$\|V^*_\lambda - V^*\|_\infty \leq \frac{\lambda}{1 - \gamma} \log |A|$$

of the true (unregularized) optimum (Li et al., 2019). However, recent results indicate an exponential decay rate of the regularization error in $\lambda$:

$$V^*(s) - V^{\pi^*_\lambda}(s) \leq C_3 \exp(-C_4/\lambda + O(\log \lambda))$$

where $C_3$, $C_4$ are problem-dependent, and a matching lower bound holds up to $\lambda$-polynomial factors (Müller et al., 2024).

• Policy Iteration Convergence: Monotonic improvement guarantees that regularized policy iteration converges to the softmax optimum (Li et al., 2019).
• Implicit Bias: As $\lambda \rightarrow 0^+$, $\pi^*_\lambda$ converges not to an arbitrary optimal policy, but to the maximum-entropy optimizer among all unregularized optima, with an explicit KL-implicit bias (Müller et al., 2024).

4. Connections to General Regularized MDPs, Robustness, and Constrained MDPs

Entropy regularization is a special case of the general regularized MDP framework, where the regularizer can be any strictly concave function $\phi(\pi(\cdot|s))$; this includes Tsallis, KL, or other divergences (Geist et al., 2019, Mai et al., 2020). General $\phi$ allow for control over policy sparsity and multi-modality; e.g., Tsallis regularization can induce sparse policies, whereas the Shannon case always yields full-support policies (Li et al., 2019).

Entropy-regularized MDPs are equivalent to stochastic MDPs with Gumbel rewards, demonstrating a deep connection to distributional and robust RL frameworks (Mai et al., 2020). The regularizer's convex-analytic duality corresponds to "ambiguity sets" in robust MDPs and connects to trust-region or divergence-constrained policy updates (Mai et al., 2020, Mai et al., 2021).

In constrained MDPs, entropy-regularization smooths the Lagrangian landscape: the dual function becomes $L$-smooth and strictly concave, supporting accelerated dual-gradient descent with global $\widetilde O(1/T)$ convergence rates, and linear convergence in the single-constraint case (Ying et al., 2021). This smoothing effect is absent in unconstrained MDPs, explaining the significant acceleration seen in constrained settings.

5. Algorithmic Schemes and Implementation

Entropy-regularized dynamic programming provides analytic policy update steps via softmaxes, facilitating scalable and stable algorithm design. The canonical Soft Actor-Critic (SAC) algorithm (Li et al., 2019) applies automatic differentiation over entropy-augmented targets and policy losses. The general template involves:

1. Q-network update: Fit soft Q-values against entropy-augmented bootstrapped targets.
2. Policy update: Optimize the policy network to maximize expected soft Q-values plus an entropy bonus.
3. Target network update: Polyak or exponential moving averages of the Q-networks to stabilize training.

(Generic pseudocode is detailed in (Li et al., 2019).)

For large or continuous spaces, scalable stochastic approximations such as multilevel Monte Carlo (MLMC) can estimate the necessary soft Bellman operators with near-optimal sample complexity, independent of underlying state-action cardinality (Meunier et al., 27 Mar 2025).

Natural policy gradient (NPG) methods with entropy-regularization enjoy persistence of excitation, sublinear $\widetilde O(1/T)$, and under mild regularity, linear convergence rates for policy optimization—even with linear function approximation (Cayci et al., 2021). Continuous-time natural (Fisher–Rao) gradient flows exhibit exponential convergence globally in non-compact settings (Kerimkulov et al., 2023, Müller et al., 2024).

6. Extensions: Mutual Information, Robustness, and Partial Observability

Entropy regularization can be generalized to mutual-information regularization between states and actions, yielding adaptive reference policies and flexible exploration-exploitation trade-offs (Leibfried et al., 2019). MIRACLE algorithms implement these principles with adaptive marginal priors and outperform soft actor-critic in benchmark domains.

Robust entropy-regularized MDPs extend the framework to uncertainty in transition probabilities. The robust soft Bellman operator is

$$(T^{rob} V)(s) = \max_{\pi(\cdot|s)} \sum_{a} \pi(a|s)\big[r(s,a) + \gamma \delta(s,a;V)\big] + \tau H(\pi(\cdot|s)),$$

with $$\delta(s,a;V) = \min_{P' \in \mathcal{P}(s,a)} \mathbb{E}_{s' \sim P'}[V(s')]$$. The theory guarantees tractability, algorithmic generalizability, and explicit inner-solver complexity bounds (Mai et al., 2021).

Entropy regularization is also effective in partially observed MDPs (POMDPs) and interval MDPs (IMDPs), where it controls uncertainty and induces predictability or unpredictability in the controlled process. In entropy-regularized IMDPs, the value-function recursion involves convex optimization and achieves deterministic optimal policies, balancing cost and path entropy even under adversarial transition uncertainty (Molloy et al., 2021, Zutphen et al., 2024).

7. Comparative Perspectives and Open Problems

Entropy-regularized MDPs are strictly subsumed by general regularized and robust MDP frameworks; every (strictly concave) regularizer and ambiguity set can be associated via Legendre-Fenchel duality (Mai et al., 2020). The Shannon regularizer yields always full-support policies, whereas alternatives (e.g., Tsallis) can enforce sparsity. The bias-variance and error propagation analyses remain active research areas, with sharp exponential regularization error rates recently established (Müller et al., 2024).

Algorithmically, accelerating primal-dual methods via natural-gradient preconditioning and interplay with mirror descent significantly improves practical and theoretical convergence rates (Li et al., 2022, Kerimkulov et al., 2023).

The extension of the entropy-regularized framework to general convex potentials, action-state entropy mixtures (Grytskyy et al., 2023), and distributional, constraint-aware, or continuous-control domains remains an intensive and mature area of exploration.

---

References

• (Li et al., 2019)
• (Geist et al., 2019)
• (Neu et al., 2017)
• (Mai et al., 2020)
• (Müller et al., 2024)
• (Ying et al., 2021)
• (Mai et al., 2021)
• (Cayci et al., 2021)
• (Meunier et al., 27 Mar 2025)
• (Kerimkulov et al., 2023)
• (Molloy et al., 2021)
• (Zutphen et al., 2024)
• (Leibfried et al., 2019)
• (Li et al., 2022)
• (Grytskyy et al., 2023)