Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL

Abstract: Reinforcement learning post-training dramatically improves LLM reasoning, but suffers from training instability and diversity collapse. Advantage functions offer an appealing fix: they reshape the training objective, reweight which rollouts drive learning, and are trivial to implement. Yet a proliferation of methods makes it unclear which advantage to use and when. We cut through the confusion with a unifying framework that decomposes any advantage into its positive and negative gradient mass along two orthogonal axes. On the sign axis, imbalanced updates collapse either entropy or weight geometry. On the difficulty axis, hard-problem focus sharpens signal but costs sample size. Both trade-offs shift during training: exploration favors balance and hard focus; exploitation favors suppression and medium focus. This motivates FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage that reads training dynamics to schedule the gradient weight automatically. FADE reaches peak pass@1 20k steps earlier than the best static baseline at the 7B scale and 2k steps earlier at the 32B , while achieving the best accuracy-diversity trade-off across all pass@k on LiveCodeBench and AIME.

• The paper introduces a decomposition framework that splits policy gradients into positive and negative masses along sign and difficulty axes.
• The FADE controller dynamically schedules policy weights based on empirical signals, balancing exploration and exploitation for robust learning.
• Empirical results show that FADE achieves faster convergence and superior accuracy-diversity trade-offs compared to static baselines.

Decomposing Policy Gradient Weights in Reinforcement Learning: An Analysis of Trade-offs and Scheduling with FADE

Introduction

The paper "Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL" (2607.01490) presents a comprehensive framework for analyzing and designing policy gradient weights—referred to as "policy weights" or "advantage functions"—in LLM reinforcement learning post-training. The motivation is twofold: dramatic improvements in LLM reasoning capabilities via RL are limited by instability and diversity collapse, while a proliferation of alternative advantage functions complicates principled choice and understanding of trade-offs. This work provides a decomposition of policy weights into positive and negative gradient mass along two axes—sign and difficulty—systematically characterizing their effect on entropy, learning dynamics, and diversity. Building on these insights, the authors introduce FADE (Focal Advantage with Dynamic Entropy), a controller that adaptively schedules policy weights based on empirical training signals.

Policy Weight Decomposition: Theoretical Framework

The core analytical tool is a decomposition that splits policy gradient updates (under binary terminal rewards) into contributions from successful ($m_S$) and failed ($m_F$) trajectories. This decomposition reveals three primary axes along which advantage functions differ:

• Sign axis: The balance of positive ($m_S$) and negative ($m_F$) gradient mass. Imbalances induce either entropy (mode) collapse (when $m_S > m_F$) or rank-1 update collapse (when $m_F > m_S$).
• Difficulty axis: The allocation of gradient mass across batches with varying solve rates $p$; focusing on harder problems increases information content but also variance.
• Scale axis: The overall magnitude of the policy gradient, affecting implicit learning rate.

This decomposition is instantiated for numerous popular and recent RLVL objectives, showing which design axis each modifies (Figure 1).

Figure 1: Effect of Policy Weights on Batch Level Gradients based on the estimated solve rate $p$; each method trades off sign, difficulty, and scale axes differently.

Implications of Each Axis

• Sign Imbalance: Success-biased weighting (upweighting $m_S$) accelerates entropy collapse, leading to a high-precision, narrow policy; excessive failure bias collapses the update to rank-1, stalling learning and damaging diversity (Figures 3, 4, 5).
• Difficulty Focus: Focusing on harder problems sharpens the update, but at the expense of sample size and estimator variance. The optimal focus shifts during training as the underlying distribution of solve rates changes (Figures 6, 10, 11).
• Adaptive Policy Weighting: Static advantages cannot optimally trade off between exploration (diverse, information-rich updates) and exploitation (rapid reward maximization); thus, a dynamic approach is required.

Empirical Analysis: Instability, Diversity, and the Rank-1 Collapse

Experiments compare static and dynamic advantage functions on reasoning-capable LLMs (Qwen 2.5 7B, CWM 32B) over competitive programming (LiveCodeBench) and math (AIME) using PPO. The learning dynamics are dissected along the axes defined above.

Key findings:

• Sign-biased methods (e.g., AsymGRPO with $\delta < 1$): Yield rapid initial gains in pass@$m_F$0, but updates become nearly rank-1, with the model's output head dominating the change (up to 90% of $m_F$1 at 7B scale). Learning rapidly saturates, diversity collapses, and generalization degrades.
• Difficulty-focused methods (e.g., Power-$m_F$2): Assigning more gradient mass to hard prompts improves informative gradient content, but as training progresses and more problems are solved, this focus excessively thins the effective sample size, increasing gradient variance.
• All static tuning schemes are suboptimal as the optimal trade-off shifts with solve rate and entropy.

  Figure 2: Scaling negative gradients of mean GRPO shows that upweighting failure gradients ($m_F$3) increases correlation among correct samples but makes failure gradients highly decorrelated, leading to rank collapse.

  

  Figure 3: Policy entropy is tightly controlled by the sign ratio $m_F$4, rather than directly by pass@$m_F$5 or learning speed.

  

Figure 4: Failure-biased methods precipitously collapse the effective update dimension (as measured by singular-value ratios), confirming empirical rank-1 collapse.

FADE: Focal Advantage with Dynamic Entropy

To resolve these intrinsic trade-offs, the authors introduce FADE: a policy weight controller that adapts dynamically to the training dynamics by reading empirical solve rates ($m_F$6) and entropy ($m_F$7):

• $m_F$8 (difficulty focus): Decays from high (hard-problem focus) to low as the model becomes more exploitative, modulated by the current average solve rate.
• $m_F$9 (sign balance): Decays from balanced ($m_S$0) to failure-biased as entropy falls below a target threshold $m_S$1.

The controller thus "delays exploitation," maintaining diverse, information-rich policy updates early on, before specializing as training plateaus.

Figure 5: FADE achieves superior accuracy (pass@$m_S$2) and diversity (pass@$m_S$3) faster than all static baselines across model scales on LiveCodeBench v6.

Figure 6: FADE controller dynamics: schedules $m_S$4 and $m_S$5 in response to observed entropy and solve rates, coordinating exploration (diversity) and exploitation (efficiency).

An ablation reveals that both difficulty and sign axes must be adaptively and independently scheduled; naive alternatives degrade diversity or accelerate premature convergence (Figure 7).

Figure 7: Ablation study: removing dynamic scheduling of either $m_S$6 (difficulty) or $m_S$7 (sign) impairs diversity or induces entropy collapse.

Quantitative Results

FADE's dynamic scheduling achieves strong numerical results:

• Training Efficiency: Reaches peak pass@$m_S$8 20k steps earlier than the best static baseline at 7B scale, and 2k steps earlier at 32B, for both code and math domains.
• Accuracy-Diversity Trade-off: Achieves the highest pass@$m_S$9 for all $m_F$0 and maintains flatter pass@$m_F$1 curves (i.e., sustained diversity), as demonstrated by shifted power law fits to empirical curves (Figures 6, 10, 11, 12).
• Generalization: Outperforms static methods on out-of-distribution math evaluation (AIME 2024/2025), indicating improved generalization beyond the training set.

  Figure 8: Policy weight families reveal trade-offs between accuracy and diversity; FADE navigates to the best overall region, improving both as training progresses.

  

Figure 9: Shifted power-law fits across budgets quantify the evolving diversity under FADE and baseline methods.

Figure 10: Shifted Power Law predicts pass@$m_F$2 across difficulty splits; FADE’s empirical curves reflect enhanced scaling at all difficulties.

Figure 11: Evolution of the $m_F$3 vs. pass@$m_F$4 curve during RL demonstrates FADE’s ability to maintain diversity throughout training.

Theoretical and Practical Implications

Theoretical Implications:

• The decomposition framework unifies disparate advantage design choices, enabling principled tuning and analysis.
• Identifies that both failure-driven rank-1 collapse and success-driven entropy collapse are natural, emergent dynamics of sign-imbalanced advantage functions.
• Connects batch-level difficulty focus (e.g., Power-$m_F$5) to Monte Carlo effective sample size and the scaling law of pass@$m_F$6 diversity, providing a formal basis for advantage scheduling.
• The analysis of update rank provides insight into feature space collapse and specialization in large neural policies post-RL.

Practical Implications:

• FADE provides a drop-in, self-tuning scheduler for LLM RL post-training, avoiding mode collapse and maintaining high-variance updates essential for robust generalization.
• Outperforms both classical (e.g., GRPO) and recent (e.g., pass@$m_F$7, logmeanexp) methods in both convergence speed and final performance at multiple model scales and tasks.

  Figure 12: Lower scale of gradient magnitude preserves entropy; improper scaling (e.g., via mismatched normalization) accelerates collapse.

Future Directions

Several open questions and extensions are explicitly delineated:

• Reward Granularity: Extending the $m_F$8 decomposition beyond binary terminal rewards to dense or trajectory-level reward models to assess if sign/difficulty dynamics persist at finer granularity.
• Estimation Robustness: Investigating which policy weight schemes are most stable under low rollout or high-variance solve rate estimation.
• Generalization to Multi-turn / Agentic Settings: Exploring how these dynamics manifest in multi-turn, interactive, or tool-augmented RL environments.
• Adaptation in the Low-Data Regime: Characterizing optimal FADE-style scheduling when the number of rollouts is severely limited.

Conclusion

This work provides a unified theoretical and empirical analysis of policy gradient weighting in RL fine-tuning for reasoning LLMs, exposing the inherent trade-offs between speed, diversity, and sample efficiency. By introducing FADE—a controller that adaptively regulates exploration and exploitation through dynamic scheduling of both sign and difficulty axes—the work achieves faster, more robust learning with better accuracy-diversity balance than existing methods. The analysis and FADE scheduler provide both actionable algorithms and foundational principles for future advancements in fine-tuning large autoregressive models via RL.