Gradient-Bounded Policy Optimization (GBPO)

• Gradient-Bounded Policy Optimization (GBPO) is a reinforcement learning method that directly bounds per-sample policy gradient coefficients to ensure numerical stability and reduced variance.
• It combines the safety of trust-region methods with the simplicity of first-order techniques, offering robust, stable updates in continuous-control and recommendation settings.
• Empirical results show faster convergence, higher policy entropy, and improved exploration compared to standard methods like PPO and TRPO.

Gradient Bounded Policy Optimization (GBPO) is a class of policy optimization algorithms that generalize the principle of constraining the magnitude of policy gradients to promote robust, stable, and variance-reduced reinforcement learning (RL). The approach seeks to blend the safety and monotonicity properties of trust-region methods (e.g., TRPO) with the computational and implementation simplicity of first-order, minibatch-friendly methods such as PPO, by directly bounding gradient update coefficients per sample. GBPO has been realized in various forms, including Clipped-Objective Policy Gradients (COPG) for continuous-control RL (Markowitz et al., 2023) and in large-scale sequence-level generative recommender models (Xie et al., 29 Jan 2026).

1. Motivation and Problem Statement

Numerous policy gradient algorithms—most notably TRPO and PPO—employ various mechanisms to constrain the divergence between the updated and prior policies to ensure that learning proceeds monotonicly and to prevent catastrophic policy updates. PPO achieves this by clipping the importance weight in its surrogate objective, which implicitly bounds the expected change in policy. However, PPO’s clipping is performed on the importance ratio $$r_\theta(s,a) = \frac{\pi_\theta(a|s)}{\pi_{\theta_\text{old}}(a|s)}$$, not on the gradient coefficient itself.

GBPO, as exemplified by COPG, modifies this paradigm by directly bounding the per-sample policy gradient coefficient, i.e., the factor by which the baseline policy gradient $\nabla_\theta \log \pi_\theta(a|s) \, \hat{A}$ is scaled, replacing surrogate loss-based clipping with explicit, gradient-level bounds (Markowitz et al., 2023). This mechanism generalizes naturally to list-wise RLHF recommendation settings, where it yields both improved stability and unique bias characteristics (Xie et al., 29 Jan 2026).

2. Core Formulation and Algorithmic Structure

The central GBPO update at the sample level constrains the policy gradient coefficient to the interval $[1-\epsilon,1+\epsilon]$ (or a function thereof), where $\epsilon$ is a tunable parameter. In COPG, the canonical instantiation, the objective is: $$J_{\text{COPG}}(\theta) = \mathbb{E}_t\left[\min\left(\log \pi_\theta(a_t|s_t)\hat{A}_t, \log(\text{clip}(r_\theta(s_t,a_t), 1-\epsilon, 1+\epsilon)\;\pi_{\theta_\text{old}}(a_t|s_t))\hat{A}_t\right)\right]$$ where the clipping is performed element-wise with respect to $\hat{A}_t$'s sign. The gradient estimate can be viewed as

$$\nabla_{\theta} J_{\text{COPG}} = \begin{cases} \frac{\pi_{\theta_{\mathrm{old}}}}{\pi_\theta}\nabla_{\theta} J_{\text{PPO}} & \text{(no clip)} \ \frac{1}{1\pm\epsilon}\nabla_{\theta} J_{\text{PPO}} & \text{(clipped)} \end{cases}$$

(Markowitz et al., 2023). The procedure thus bounds the update contribution per trajectory, enforcing local Lipschitz continuity.

In the sequence modeling context (e.g., OneRec-V2 (Xie et al., 29 Jan 2026)), GBPO generalizes to slates $S$, with the sequence-level importance ratio defined as the geometric mean over log-probabilities: $$r_{\text{slate}}(\theta) = \exp\left[\frac{1}{L} \sum_{t=1}^L \log\frac{\pi_\theta(i_t|u,S_{<t})}{\pi_{\theta_{\text{old}}}(i_t|u,S_{<t})}\right]$$ The static and symmetric clipping is then

$$\Phi_{\text{static}}(r) = \max(r,1)\implies \frac{r}{\Phi_{\text{static}}(r)} = \min(r,1)$$

so positive advantage updates above $r=1$ are clipped.

3. Bias–Variance Properties and Exploration Implications

Strict bounding of gradient coefficients induces bias in the policy gradient estimator, as it removes the full importance-weighted correction present in unbiased off-policy learning. Both PPO and GBPO use biased gradient estimates, but the form of bias differs. PPO's clipping suppresses large positive-advantage updates while amplifying negative-advantage steps; GBPO's direct bounding "tilts" this effect further:

• For $\hat{A} > 0$, the multiplier is typically $1/(1+\epsilon)<1$, producing smaller steps into high-value regions.
• For $\hat{A}<0$, the multiplier is $1/(1-\epsilon) > 1$, yielding larger ejection from low-value zones.

This pessimistic bias empirically raises policy entropy, combating premature collapse into sub-optimal deterministic policies and improving exploration (Markowitz et al., 2023). The variance is reduced, quantified to be $$\mathcal{O}(\mathrm{Var}[\nabla_\theta \log \pi\; \hat{A}])$$ per step—eliminating the exponential variance growth associated with deeper products of importance weights.

4. Empirical Results and Comparative Performance

COPG (as an instance of GBPO) demonstrated consistently superior or competitive performance versus PPO and TRPO on continuous-control MuJoCo benchmarks, Safety Gym tasks (both unconstrained and in reward-constrained RCPO mode), and Multi-Task Meta-World MT10. Empirical highlights include:

• Final average returns exceeding PPO by $5$–$10\%$.
• Faster convergence and higher policy entropy.
• Comparable or superior attainment of reward/cost constraints relative to TRPO.
• Greater robustness across seeds and smoother learning curves (Markowitz et al., 2023).

In generative recommendation, OneRec-V2's GBPO stabilized large-scale RLHF training but at the cost of suppressed cold-start discovery (cold-start video views dropped by $44.7\%$) and reduced diversity (cluster density up $11.7\%$), pointing to the adverse "Symmetric Conservatism" bias of static bounds (Xie et al., 29 Jan 2026).

5. Critique: Limitations and the Symmetric Conservatism Problem

While gradient bounding ensures numerical stability and variance control, the imposition of static, symmetric boundaries has distinct shortcomings. Specifically, GBPO as deployed in OneRec-V2 imposes the same ceiling on positive updates for both high-probability (frequent) and low-probability (cold) items. This suppresses adaptive amplification of rare, but valuable, cold-start signals, inhibiting the model's ability to promote newly emerging items into the recommendation set. Additionally, symmetric penalization prevents dynamic adjustment in response to feedback diversity, a flaw that is especially consequential in high-noise environments: GBPO incentivizes collapse toward a narrow set of "safe" popular entities, diminishing intra-list entropy and reducing overall diversity (Xie et al., 29 Jan 2026).

6. Extensions and Adaptive Variants

Subsequent to the observation of GBPO's "Symmetric Conservatism," adaptive gradient bounding has been proposed—for example, SAGE (Sequence-level Adaptive Gradient Evolution) (Xie et al., 29 Jan 2026), which introduces:

• Asymmetric Adaptive Bounds: Positive updates ($A\ge0$) can receive a "boost factor" for $r_\text{slate}>1+\epsilon_\text{boost}$, permitting super-linear amplification of cold-start items.
• Entropy-aware Penalties: Negative updates ($A<0$) are modulated by the entropy deficit relative to a running mean, selectively penalizing collapsed slates while preserving exploratory rejected slates.

Empirically, these innovations restore cold-start recall (+43--63%), increase list entropy (+6--10.4%), and globally improve NDCG@10 by up to $8.9\%$ compared to vanilla GBPO in RLHF for sequence recommendation (Xie et al., 29 Jan 2026).

7. Algorithmic Implementation and Practical Recommendations

The implementation of GBPO is architecturally analogous to PPO. The primary distinction is the objective function and its corresponding per-sample coefficient bounding. In canonical deep RL, the procedure is (Algorithm 1, (Markowitz et al., 2023)):

• Collect trajectories under $\pi_{\theta_\text{old}}$.
• For $K$ epochs and each minibatch $B$:
  • Compute $r(s,a)$.
  • Compute unclipped and clipped gradients $g_1, g_2$.
  • Set $g = \min(g_1,g_2)$ (element-wise by sign of advantage).
  • Update parameters: $\theta \leftarrow \theta + \alpha \mathbb{E}_B[g]$.
• Update $\theta_\text{old} \leftarrow \theta$.

For sequence-level GBPO, implementation is identical in structure, with computation performed at the slate level, leveraging the geometric mean of per-token likelihood ratios (Xie et al., 29 Jan 2026). Recommended $\epsilon$ values are $0.1$–$0.3$, with PPO’s default $\epsilon=0.2$ proving effective; annealing $\epsilon$ over training is optional.

The method maintains theoretical guarantees of $O(1)$ coefficient bounds, thereby precluding gradient explosion or vanishing, and achieves excellent robustness, generalizability, and exploratory capacity across both classic RL and large-scale generative recommendation settings (Markowitz et al., 2023, Xie et al., 29 Jan 2026).