Bounded Action Space Gradients

• Bounded Action Space Gradients are methods in reinforcement learning that adjust policy gradients to respect strict action limits, addressing bias and variance.
• They employ techniques like Beta policies, action clipping, and marginalization to align learning signals with the true bounded support of actions.
• These approaches enhance convergence, stability, and sample efficiency in continuous control tasks across robotics, simulations, and planning applications.

Bounded action space gradients are a class of methods and estimators developed for reinforcement learning (RL) and planning in continuous control domains where actions are restricted to lie within a bounded domain, typically a box or a sphere in $\mathbb{R}^d$. Standard policy gradient approaches, often built upon Gaussian or unbounded policies, fail to align with the real bounded support of actions in such tasks, producing estimation bias, inflated variance, and suboptimal learning dynamics. Addressing this mismatch, bounded action space gradient techniques—encompassing reparameterization with bounded-support distributions, variance reduction via marginalization, and domain-specific estimators—formally align the learning signal and the true action space, eliminate boundary effects, and enable stable, sample-efficient optimization in practical RL problems.

1. Motivation: The Bounded Action Space Problem

Many RL environments, such as robotics and simulated control benchmarks, enforce explicit bounds on actions for safety, physical plausibility, or hardware constraints. Conventional policy gradient methods frequently parameterize stochastic policies as Gaussians over $\mathbb{R}^d$, then clip or squash the sampled actions before execution. This usage of an unbounded prior with an implicit post-sampling bound introduces bias and variance through several mechanisms:

• Estimation bias at the boundary: The Gaussian's nonzero probability mass outside the legal range cannot be directly exploited, leading to "boundary bias" in the likelihood ratio.
• Mismatched score functions: The gradient $\nabla_\theta \log \pi_\theta(a\mid s)$, as applied naively, is not valid when $a$ is subjected to a nonlinear projection or transformation.
• Variance inflation: Out-of-bound samples are mapped to the boundary, producing constant $Q$-values and higher gradient estimator variance.

This gap between the learned policy's support and the environment's action constraints can degrade convergence speed, achievable reward, and empirical stability (Petrazzini et al., 2021, Fujita et al., 2018, Eisenach et al., 2018).

2. Principled Construction of Policy Gradients under Bounds

Recent work has established a variety of principled estimators and algorithms that ensure the policy gradient is consistent with the environment's bounded action support.

2.1 Beta Policies and Bounded-Support Parameterizations

The policy can be directly parameterized using bounded-support distributions, such as the Beta family for $[a_\text{min},a_\text{max}]$ actions:

• Rescaling: Actions $a\in[a_\text{min},a_\text{max}]$ are rescaled to $x\in[0,1]$ via $$x = (a - a_\text{min}) / (a_\text{max} - a_\text{min})$$.
• Beta policy: $$\pi_\theta(a\mid s) = \mathrm{Beta}\big(x; \alpha(s;\theta),\beta(s;\theta)\big)/(a_\text{max} - a_\text{min})$$, with $\alpha,\ \beta > 1$ for unimodality (Petrazzini et al., 2021).
• Log-probability and gradient: Analytical expressions yield correct $\log\pi_\theta(a\mid s)$ and parameter gradients, using digamma terms for $\alpha$ and $\beta$, and chain rule to backpropagate through the network.

Replacing a Gaussian with a Beta in PPO eliminates boundary bias and empirically improves learning stability, convergence speed, and final reward, as shown on OpenAI Gym tasks (Petrazzini et al., 2021).

2.2 Marginal and Clipped Policy Gradients (MPG, CAPG)

Marginal Policy Gradient (MPG) estimators exploit the fact that after any transformation $T$ that projects unbounded samples $u$ into bounded actions $a = T(u)$, only the marginal distribution over the final $a$ is relevant.

Clipped Action Policy Gradient (CAPG)

• Action Clipping: For $a\in [a_\text{min},a_\text{max}]$, $c(u) = \min(\max(u,a_\text{min}), a_\text{max})$.
• Variance-reduced estimator: Compute the gradient not with $\nabla_\theta \log \pi_\theta(u|s)$ for all $u$, but with specially constructed "clipped scores"—constants in the out-of-bounds regions, analytical in-bounds (Fujita et al., 2018, Eisenach et al., 2018).
• Variance reduction: Provably lower variance estimator for the same unbiasedness, through Rao–Blackwellization over the original unbounded variable.

Angular Policy Gradient (APG) and General MPG

• Directional bounds: For unit spheres and other non-box constraints, similar marginalization applies (Eisenach et al., 2018).
• Unified analysis: For any mapping $T$ from $\mathbb{R}^d$ to bounded $\mathcal{A}$, the variance reduction property holds.

2.3 Expected Policy Gradients (EPG) and All-Action Methods

Alternative approaches directly integrate (analytically or numerically) over the bounded action space rather than sampling, yielding the expected value of the policy gradient estimator (Ciosek et al., 2018, Petit et al., 2019). These methods encompass both stochastic and deterministic policy gradients and can use analytic forms or Monte Carlo quadrature as required.

3. Algorithmic Implementations and Practical Schemes

The transition from principle to algorithm involves both architectural and estimator choices.

• Network output post-processing: For Beta policies, map network outputs through $1+\mathrm{softplus}$ to ensure $\alpha,\beta > 1$ for unimodality and numerical stability (Petrazzini et al., 2021).
• Surrogate objectives in PPO: Clipping ratios, entropy bonuses, early-stopping on KL divergence, and weight decay are retained as with Gaussian policies, but with correctly computed log-likelihoods and gradients.
• Numerical stability: Gradient computation with respect to the Beta shape parameters leverages automatic differentiation; avoid direct instantiation outside the support.

A generalized implementation sketch (PyTorch-style) appears in (Petrazzini et al., 2021), covering log-probability computation, ratio calculation, clipped PPO loss assembly, entropy, and optimization.

Empirical hyperparameter choices remain similar, with adjustments to learning rates, batch sizes, and network widths specified for each environment.

4. Theoretical Properties: Bias, Variance, and Convergence

4.1 Bias Elimination and Score Boundedness

• Boundary Bias: Truncated or projected policies (e.g., Gaussian with clipping) naturally induce bias that can only be eliminated with full support parameterizations or explicit variance-reduced estimators. Notably, Gaussian policies violate the bounded score assumption, introducing an irreducible persistent bias $B(R) = O(1/R)$ unless the radius $R\to \infty$, which is incompatible with actual bounded actions (Bedi et al., 2022).
• Heavy-tailed policies: Student-t parameterizations can provide bounded score functions and eliminate inverse-radius bias at the cost of potentially increased gradient variance (Bedi et al., 2022).

4.2 Variance Reduction via Marginalization

All-action, marginal, CAPG, and APG estimators guarantee a reduction in gradient estimate variance by integrating away irrelevant dimensions or replacing high-variance scores in regions where the environment's response is constant. This effect is quantifiable through Fisher information and law of total variance (Eisenach et al., 2018, Fujita et al., 2018, Petit et al., 2019).

4.3 Convergence and Sample Efficiency

• All-action/EPG approaches: Integration across the bounded action domain yields zero variance for locally quadratic critics and analytic policies, providing maximally efficient gradients in those settings (Ciosek et al., 2018, Petit et al., 2019).
• Stability: Empirically, Beta policies, CAPG, and EPG deliver more reliable and faster convergence on continuous control tasks relative to conventional (clipped) Gaussian policies (Petrazzini et al., 2021, Fujita et al., 2018, Ciosek et al., 2018).

5. Empirical Performance and Benchmark Results

• OpenAI Gym tasks: Proximal Policy Optimization (PPO) with Beta policy yielded deterministically higher final reward and reduced variance compared to PPO with Gaussian (e.g., LunarLanderContinuous-v2 mean reward improved from ≈225.7 to ≈267.0, σ reduced from ≈50 to ≈10) (Petrazzini et al., 2021).
• CarRacing-v0 (image-based): PPO+Beta boosted the stochastic policy's success rate from ≈62% (Gaussian) to 100% (Beta), a 63% improvement (Petrazzini et al., 2021).
• Continuous control (MuJoCo): CAPG and all-action methods consistently outperformed standard estimators in sample efficiency and learning speed (Fujita et al., 2018, Petit et al., 2019).
• Bandit tasks: CAPG reduced per-sample $\theta$-gradient standard deviation by up to 30–50% (Fujita et al., 2018).
• Variance scaling: All-action gradient estimator variance falls as $O(1/N_s)$ with integration samples and plateaus at critic MSE (Petit et al., 2019).

6. Extensions, Limitations, and Open Questions

• Assumptions: Many methods assume the policy has either independent action components (for closed-form CDF/computation) or that the critic is amenable to analytic moment computation (for EPG).
• Policy choice: Open questions remain on the optimal bounded-support policy family—preliminary results suggest Beta, truncated Gaussian, or Student-t policies, but trade-offs exist in stability, entropy, and computational cost (Petrazzini et al., 2021, Bedi et al., 2022).
• Action-dependent variance: As action dimension increases, quadrature methods become computationally infeasible; Monte Carlo integration or analytic marginalization are preferred (Petit et al., 2019, Ciosek et al., 2018).
• Heavy-tailed/stability trade-off: While Student-t policies eliminate bias, they may require batch size tuning and stability regularization (Bedi et al., 2022).
• Structure of $Q(s,a)$: High regularity or locally quadratic critics enable analytic integration; otherwise, sample-based or approximate techniques are necessary (Ciosek et al., 2018).

7. Relationships to Planning and Tree Search with Bounded Actions

Action-gradient techniques have been extended to online planning, including search in continuous MDPs and POMDPs:

• The action-gradient theorem derived for bounded spaces enables direct gradient-based optimization of action values in tree search algorithms, with proper handling via projection or reparameterization to maintain feasibility (Lev-Yehudi et al., 15 Mar 2025).
• Multiple Importance Sampling (MIS) trees facilitate efficient sample reuse across branch updates in bounded action domains, improving optimization performance in high-dimensional or uncertain environments.

These developments mark the extension of bounded action space gradient methodology from RL to model-based planning and search, preserving estimator stability and efficiency across a wider spectrum of control and planning settings.