Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works

Abstract: Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards, but its group-mean-centered advantage can fail under binary rewards. The failure mode is gradient starvation: when every response in a group is correct or every response is wrong, the centered advantage is exactly zero and the policy receives no learning signal. We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality, and observe a 0.69 degeneracy rate at group size four in logged Qwen3.5-9B GSM8K training. We then show that the fixed-reference Sign advantage, $A=2r-1$, performs pass@$G$ failure descent by increasing the probability that at least one sample in the group succeeds. On the full GSM8K test set across seven seeds, Sign reaches 73.8% accuracy versus 28.4% for standard normalized group-mean DrGRPO at group size four, a 45.4 point gain with $p<0.0001$. The effect is directionally consistent on Llama-3.1-8B and positive but underpowered on a MATH-500 transfer check. Pass@$k$ analysis indicates that the main benefit is search compression rather than large capacity expansion, aligning the empirical gains with recent RLVR ceiling observations.

• The paper finds that group-mean centering in GRPO fails in binary reward settings due to gradient starvation in degenerate groups.
• It demonstrates that a fixed-reference Sign advantage guarantees nonzero gradients, enabling reliable improvement in pass@k metrics.
• Empirical results reveal that the Sign advantage can boost accuracy by up to 45.4 percentage points over traditional normalization methods.

Gradient Starvation in Binary-Reward GRPO: Diagnosing the Failure of Group-Mean Centering and the Efficacy of Fixed-Reference Advantages

Introduction

Group Relative Policy Optimization (GRPO) has emerged as a prominent reinforcement learning methodology for LLM finetuning in the presence of verifiable (binary) rewards—especially in math and reasoning tasks. This approach constructs group-relative advantages by generating multiple responses per prompt, scoring them, and centering policy updates relative to the group mean. However, when rewards are binary, standard group-mean centering becomes susceptible to a debilitating phenomenon: gradient starvation. This essay provides a comprehensive technical synthesis of the mechanisms underlying gradient starvation, the rigorous empirical and theoretical evidence quantifying its real impact, and a critical analysis of the simplest and most effective remedy: switching to a fixed-reference, "Sign" advantage. The implications for RLVR algorithm design are profound, particularly in sample-efficient or low-compute regimes.

Mechanism of Gradient Starvation Under Group-Mean Centering

In GRPO with binary rewards $(r \in \{0,1\})$, group-mean centering computes advantages $A_j = r_j - \bar{r}$. In groups where all responses are either correct or incorrect (degenerate groups), $\bar{r}$ equals each $r_j$, yielding zero advantage for all responses, and thus, zero policy gradient. This degeneracy is frequent when per-prompt success probabilities $p_x$ approach $0$ or $1$. The critical insight is that, due to the convexity of the degeneracy function $D(p_x, G) = p_x^G + (1 - p_x)^G$ and per-prompt bimodality, the real degeneracy rate significantly overshoots the i.i.d. Bernoulli prediction (Jensen gap). For instance, at $G = 4$, over 69% of groups were degenerate in GSM8K training with Qwen3.5-9B (Figure 1), confirming that the majority of training computation is wasted without providing a learning signal.

Figure 1: Left: The per-prompt success-rate distribution at training start ($G=2$) is highly bimodal, explaining the unexpectedly high observed degeneracy. Right: Observed degeneracy is 0.90, vastly exceeding the Bernoulli prediction due to bimodality in $A_j = r_j - \bar{r}$0.

This phenomenon is not merely empirical observation but is underpinned by formal justification. By Jensen's inequality, $A_j = r_j - \bar{r}$1, where the gap grows with the variance of $A_j = r_j - \bar{r}$2, leading to unexpectedly high starvation even for moderate group sizes.

Theoretical Analysis: Fixed-Reference Advantages and Pass@$A_j = r_j - \bar{r}$3 Failure Descent

To bypass gradient starvation, the paper considers fixed-reference advantage formulations, specifically the Sign advantage: $A_j = r_j - \bar{r}$4. Unlike group-mean centering, this approach always imparts nonzero gradient for all groups, regardless of their composition. The paper provides a formal theorem establishing that, for Sign and other fixed-reference advantages, the all-fail group component yields a gradient proportional to $A_j = r_j - \bar{r}$5 (where $A_j = r_j - \bar{r}$6), performing direct gradient ascent on $A_j = r_j - \bar{r}$7. Thus, training explicitly increases the probability that each group yields at least one correct answer. This is in stark contrast to group-mean centering, which provides zero signal when needed most.

Figure 2: Left: Pass@$A_j = r_j - \bar{r}$8 curves for different advantage choices illustrate that Sign delivers consistent improvements in pass@1 and modest expansion for larger $A_j = r_j - \bar{r}$9. Right: The Sign advantage outperforms alternatives at $\bar{r}$0 and $\bar{r}$1.

Notably, the effectiveness of an advantage formulation is governed not by its expected gradient magnitude across all group compositions, but by its ability to provide nonvanishing gradient in degenerate groups. Thus, coverage is the critical axis for small-group RLVR success.

Empirical Results: Dominance of Sign Advantage and Group Size Effects

Experiments with Qwen3.5-9B on GSM8K reveal a dramatic performance gap between advantage formulations at $\bar{r}$2: group-mean normalized DrGRPO achieves only 28.4% accuracy versus 73.8% for Sign, a 45.4pp gain ($\bar{r}$3). This effect is directionally consistent across model scales and families (Llama, Qwen), but vanishes for extremely weak models where $\bar{r}$4 for almost all prompts.

Figure 3: GSM8K accuracy (%) for six advantage formulations shows Sign superiority at $\bar{r}$5, with DrGRPO's performance indistinguishable from the base model under severe gradient starvation.

Interestingly, increasing group size ($\bar{r}$6) rescues group-mean centering; at $\bar{r}$7, DrGRPO rises to 81.7% mean accuracy. However, Sign remains competitive and slightly superior (85.8% at $\bar{r}$8), indicating that fixed-reference advantages offer further robustness even when group size mitigates starvation (Figure 4).

Figure 4: Left: Sign escape from low-reward gradients is rapid, while DrGRPO is stably trapped. Right: The persistent high fraction of all-fail groups explains DrGRPO's lack of progress at small $\bar{r}$9.

Pass@$r_j$0 analysis further supports the search-compression interpretation: Sign increases pass@1 most, with diminishing returns at higher $r_j$1, aligning with prior findings that RLVR primarily concentrates existing solution mass rather than expanding model capacity.

Broader Implications and Future Directions

These results compel a re-evaluation of RLVR protocol design. For binary reward settings—or any regime with extreme reward sparsity—advantage normalization must ensure nonzero signal in degenerate settings. Fixed-reference advantages are algorithmically trivial but principled solutions. Alternatively, increasing $r_j$2 or employing generation-time filtering (e.g., DAPO) can mitigate starvation, but with higher computational cost. The superiority of Sign at moderate $r_j$3 is particularly salient in scenarios constrained by budget or latency.

The exploration of cross-family and cross-dataset generalization (GSM8K, MATH-500) showed positive but limited evidence for broad applicability. Limitations include insufficient improvement for ultra-weak base models and inconsistent results on MATH-500 transfer at $r_j$4. Additional research is needed to systematically compare fixed-reference and dynamic sampling strategies, and to examine convergence properties with larger compute or alternative RLVR architectures.

Conclusion

Gradient starvation presents a substantial and previously underappreciated obstacle when using group-mean centering in GRPO with binary rewards. The degeneracy rate is underestimated by standard i.i.d. analyses due to the prevalence of hard and easy prompts, as rigorously quantified in the paper. The simplest fixed-reference advantage, Sign, robustly resolves starvation and yields substantial performance improvements in small-$r_j$5 regimes without incurring additional compute. While larger group sizes can alleviate starvation, fixed-reference advantages are theoretically sound, easy to implement, and empirically effective across models and datasets above a minimal base capability threshold. Going forward, RLVR algorithm design should prioritize degeneracy-aware advantage normalization for efficient and reliable reasoning model improvement.