Your Group-Relative Advantage Is Biased

Abstract: Reinforcement Learning from Verifier Rewards (RLVR) has emerged as a widely used approach for post-training LLMs on reasoning tasks, with group-based methods such as GRPO and its variants gaining broad adoption. These methods rely on group-relative advantage estimation to avoid learned critics, yet its theoretical properties remain poorly understood. In this work, we uncover a fundamental issue of group-based RL: the group-relative advantage estimator is inherently biased relative to the true (expected) advantage. We provide the first theoretical analysis showing that it systematically underestimates advantages for hard prompts and overestimates them for easy prompts, leading to imbalanced exploration and exploitation. To address this issue, we propose History-Aware Adaptive Difficulty Weighting (HA-DW), an adaptive reweighting scheme that adjusts advantage estimates based on an evolving difficulty anchor and training dynamics. Both theoretical analysis and experiments on five mathematical reasoning benchmarks demonstrate that HA-DW consistently improves performance when integrated into GRPO and its variants. Our results suggest that correcting biased advantage estimation is critical for robust and efficient RLVR training.

• The paper demonstrates that the common group-based advantage estimator systematically underestimates hard prompts and overestimates easy ones.
• Methodological rigor is achieved through both theoretical derivations and empirical validations on benchmarks like MATH, AMC23, and OlympiadBench.
• The proposed history-aware adaptive correction (HA-DW) effectively reweights advantages, yielding 2–4% accuracy improvements and enhanced reasoning performance.

Group-Relative Advantage Estimation Bias and History-Aware Correction

Introduction

Reinforcement Learning from Verifier Rewards (RLVR) is widely adopted for post-training LLMs on reasoning tasks. A significant trend has been the use of group-based algorithms such as Group-Relative Proximal Policy Optimization (GRPO) and its variants, which forgo value function fitting in favor of group-relative advantage estimators. These methods estimate advantage by comparing each sampled response’s reward to the intra-group empirical mean, but the statistical properties and limitations of this estimator have lacked comprehensive analysis. The paper "Your Group-Relative Advantage Is Biased" (2601.08521) rigorously exposes a systematic bias in group-based advantage estimation—demonstrating both the theoretical mechanism and empirical manifestation—and proposes a history-aware adaptive correction (HA-DW) that addresses this bias to improve RLVR outcomes.

Theoretical Analysis: Systematic Bias of Group-Relative Advantage

The analysis establishes that the standard estimator

$\hat{A}_{t,i} = r_{t,i} - \hat{p}_t$

with $\hat{p}_t = \frac{1}{G} \sum_{i=1}^{G} r_{t,i}$ acting as the baseline for the group, systematically diverges from the ground-truth advantage $A_{t,i} = r_{t,i} - p_t$ except for $p_t = 0.5$.

Specifically, the estimator:

• Underestimates advantage for hard prompts ($p_t < 0.5$),
• Overestimates advantage for easy prompts ($p_t > 0.5$),
• Is unbiased only at $p_t = 0.5$.

This is demonstrated by conditioning on the set of non-degenerate groups (those which yield nontrivial gradients, i.e., not all-correct or all-incorrect rollouts), leading to conditioning event $\mathcal{S} : 1 \le R \le G-1$. Under this conditioning, the expectation and distributional properties of $\hat{A}_{t,i}$ are derived, showing the bias is present across most of the prompt space (see Figure 1).

Figure 1: Advantage estimation bias is a monotonic function of prompt difficulty $p_t$ and grows with smaller group size $G$.

The theoretical results are further extended by closed-form and probabilistic bounds: for practical $G \leq 8$, the probability of a negative bias for hard prompts (and positive for easy ones) is over 63%, rising to $\sim$78% for extreme difficulties ($|p_t-0.5| > 0.25$) (Corollary~\ref{cor:prob_bias}). These biases persist under continuous, bounded reward distributions (e.g., Beta or truncated Gaussian) as rigorously shown in the appendices.

Empirical Characterization and Manifestation

Extensive empirical analysis on mathematical reasoning benchmarks (MATH, AMC23, AIME25, Minerva, OlympiadBench) using Qwen3-4B-Base demonstrates the practical effect: without correction, existing group-based RL algorithms plateau at suboptimal performance, particularly failing to learn from intrinsically difficult (hard) prompts due to underestimated advantages (see Figure 1a).

Figure 2: (a) Performance gains from integrating HA-DW into RL algorithms across benchmarks, (b) Direct evidence of significant estimation bias on the MATH dataset for varying group sizes, (c) Stratified improvements by difficulty level with HA-DW.

Distributional studies on rollout statistics (number of correct/incorrect responses per prompt over 8 and 128 rollouts) further validate the theoretical predictions, with lower coverage of hard prompts under small group size (see Figure 3).

Figure 3: Distributional shift in prompt statistics with increased rollouts confirms bias in advantage estimation for group-based RL.

History-Aware Adaptive Difficulty Weighting (HA-DW)

Motivated by the bias, the authors introduce HA-DW—a plug-in correction mechanism that adaptively scales the advantage for each sample based on the evolving model state and observed training dynamics (see Figure 4).

The method:

1. Evolving Difficulty Anchor: Tracks the model's moving accuracy via a dynamically updated latent belief state $C_t$, computed using a history buffer and a Kalman-style update. This anchor captures long-term reward trends and model progression, allowing the system to reference not just batch-level estimates, but a stabilized cross-batch baseline.
2. Adaptive Advantage Reweighting: Each prompt’s difficulty is estimated relative to the evolving anchor, and a scaling factor $\Phi_{t,i}$ is assigned:

$$\Phi_{t,i} = \lambda_\mathrm{scale} \cdot \exp(D_{t,i} \cdot M_t)$$

where $D_{t,i}$ determines directionality based on whether the prompt is above or below the anchor, and $M_t$ quantifies the absolute magnitude of the deviation. $\lambda_\mathrm{scale}$ is theoretically characterized to guarantee improved closeness to the true advantage (Theorem~\ref{theo:algorithm}).

Figure 4: Schematic of HA-DW. Phase one accumulates cross-batch information via an evolving anchor. Phase two adaptively reweights advantages based on deviation from the anchor.

HA-DW amplifies underweighted hard samples and suppresses overestimated easy ones, directly targeting the mechanisms responsible for learning imbalance in RLVR with limited rollouts.

Experimental Evaluation

HA-DW delivers consistent performance gains across group-based RLVR algorithms (GRPO, GSPO, DAPO) and multiple model architectures (Qwen3-4B/8B, LLaMA-3.2-3B-Instruct). Typical results show +2–4% accuracy improvement averaged across benchmarks, with largest relative gains on the hardest evaluation items (see Figure 1c, Table 1).

Ablation studies confirm:

• Dynamically evolving anchors $C_t$ outperform fixed thresholds for difficulty discrimination (Table 2).
• HA-DW can replace brute-force increases in rollouts: for instance, GRPO+HA-DW with $G=8$ outperforms vanilla GRPO with $G=16$ or higher (Table 3).
• The optimal $\lambda_\mathrm{scale}$ lies in a theoretically derived range, maximizing the correction without distorting easier sample gradients (see Table~\ref{tab:differen-lambda}).

Learning dynamics visualizations indicate more robust and higher plateau training curves, higher average reward, and longer, more intricate response generations—evidencing both stronger exploration and improved credit assignment (Figure 5).

Figure 5: Training trajectory comparison—HA-DW consistently increases final accuracy, obtained reward, and supports extended reasoning chains.

Practical and Theoretical Implications

The core finding is that group-based RL without historical difficulty correction is fundamentally misaligned: the advantage estimator introduces a training shortcut that exploits easy prompts and neglects hard ones, impeding the emergence of strong reasoning capabilities. Integrating history-aware, difficulty-adaptive reweighting is not only computationally efficient but provably corrects for bias, improving both learning stability and sample efficiency.

Related literature on alternative estimation and debiasing approaches—e.g., MAPO (Huang et al., 23 Sep 2025), LSPO (Chen et al., 1 Oct 2025), KTAE (Sun et al., 22 May 2025)—offers complementary solutions (trajectory-level, segment-level, token-level), but most do not adapt across the evolving state of the LLM or provide the same theoretical guarantees under limited rollouts. The cross-batch difficulty anchor mechanism may have applications in broader RL settings, including multi-agent, off-policy, or non-stationary environments, wherever truncated or group-relative sampling is used.

Conclusion

This work demonstrates that group-relative advantage estimation is inherently biased under typical RLVR settings for LLMs, directly suppressing learning from hard samples and overstating gains from easy ones. The proposed history-aware adaptive correction (HA-DW) offers both theoretical and empirical validation as an effective, scalable mitigation, leading to improved stability and final performance across representative reasoning benchmarks and model classes. The results emphasize that dynamic, history-aware difficulty correction should become standard in group-based policy optimization for LLM RL finetuning.

Future research should focus on generalizing such correction mechanisms to other forms of estimator bias, integrating with more advanced difficulty/artifact detectors, and optimizing historical information aggregation for context-specific RL settings.

(2601.08521)