Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control

Abstract: Reinforcement learning (RL) has unlocked complex reasoning abilities in LLMs. However, most RL algorithms suffer from performance saturation, preventing further gains as RL training scales. This problem can be characterized by the collapse of entropy, a key diagnostic for exploration in RL. Existing attempts have tried to prevent entropy collapse through regularization or clipping, but their resulting entropy curves often exhibit instability in the long term, which hinders performance gains. In this paper, we introduce Entrocraft, a simple rejection-sampling approach that realizes any user-customized entropy schedule by biasing the advantage distributions. Entrocraft requires no objective regularization and is advantage-estimator-agnostic. Theoretically, we relate per-step entropy change to the advantage distribution under minimal assumptions, which explains the behavior of existing RL and entropy-preserving methods. Entrocraft also enables a systematic study of entropy schedules, where we find that linear annealing, which starts high and decays to a slightly lower target, performs best. Empirically, Entrocraft addresses performance saturation, significantly improving generalization, output diversity, and long-term training. It enables a 4B model to outperform an 8B baseline, sustains improvement for up to 4x longer before plateauing, and raises pass@K by 50% over the baseline.

• The paper introduces Entrocraft, a method that precisely controls entropy via rejection sampling to mitigate performance saturation in LLM RL.
• Empirical results demonstrate up to a 50% improvement in pass@32 on math reasoning benchmarks and enhanced long-horizon training stability.
• The approach recasts entropy as a tunable hyperparameter, enabling a controlled exploration-exploitation balance and improved model generalization.

Precise Entropy Curve Control in LLM Reinforcement Learning: The Entrocraft Framework

Introduction

Scaling Reinforcement Learning (RL) for LLMs is fundamentally constrained by performance saturation—a regime where further training yields diminishing improvements in generalization and diversity. This phenomenon is intimately linked to the collapse of policy entropy, which reflects the explored action space during RL post-training. Existing interventions such as entropy regularization and advantage clipping fail to provide consistent, fine-grained entropy control over long horizons, often inducing further instability or failing to prevent entropy collapse.

"Addressing Performance Saturation for LLM RL via Precise Entropy Curve Control" (2604.26326) introduces Entrocraft, a methodologically minimal yet powerful entropy-control strategy for LLM RL. Entrocraft utilizes entropy-guided rejection sampling to precisely shape the entropy curve throughout training, enabling explicit entropy annealing schedules. This approach removes the need for explicit entropy-regularized objectives, is advantage-estimator-agnostic, and provides a practical mechanism for sustained gains beyond the early saturation regime.

Figure 1: Overview of Entrocraft: Entropy-guided rejection sampling is used during RL rollouts to maintain a desired entropy schedule, preventing saturation and improving generalization, diversity, and long-term training efficiency.

Theoretical Analysis of Entropy Dynamics

The paper formulates the RL entropy dynamics under realistic, non-tabular LLM policy parametrizations. The critical insight is that the expected entropy change per RL update step is negatively correlated with the empirical advantage:

• Positive-advantage samples ($\hat{A}(x, y) > 0$) make the policy more deterministic, causing entropy decrease.
• Negative-advantage samples encourage more stochasticity, increasing entropy.

Formally, entropy changes satisfy:

$\hat{A}(x, y) \cdot \Delta\mathcal{H} \leq 0$

under mild conditions on model confidence.

Figure 2: Empirical evidence that, across a variety of advantage estimators, the sufficient model confidence condition for the negative entropy-advantage correlation is strongly satisfied in practice.

This clarifies why standard RL approaches (e.g., Group Relative Policy Optimization, GRPO) experience steady entropy decay: as training progresses and the model becomes more confident, positive-advantage samples dominate, causing a monotonic drop in entropy—even when advantages are normalized. Conversely, naive attempts to stabilize entropy via regularization or clipping can over-inject noise, ultimately destabilizing learning over long horizons.

Entrocraft: Precise Entropy Curve Control via Rejection Sampling

Leveraging the theoretical link between advantage and entropy, Entrocraft frames entropy control as a problem of distributional filtering: by selectively accepting or rejecting positive- or negative-advantage rollouts conditioned on the current-batch entropy, the effective advantage distribution contributing to policy gradients can be directly shaped.

Key mechanistic details:

• During each RL step, the current minibatch entropy $\overline{\mathcal{H}}$ is compared to a target range $(h_\mathrm{low}, h_\mathrm{high})$.
• If entropy is too low (indicating collapse), positive-advantage samples are more aggressively rejected.
• If entropy is too high (indicating instability), negative-advantage samples are down-weighted or rejected.
• The acceptance probability is modulated with a temperature parameter to ensure a small degree of stochasticity and avoid dead zones.

This method’s responsiveness allows for precise tracking of an arbitrary, user-specified entropy curve. Unlike regularizers or constrained objectives, the entropy trajectory becomes a directly controllable hyperparameter analogous to a learning rate schedule.

Empirical Results

Benchmark Performance and Scaling

Entrocraft demonstrates robust and substantial improvements across a range of math reasoning benchmarks and model scales. For example, a Qwen3-4B model trained with Entrocraft outperforms a standard Qwen3-8B model, demonstrating the efficiency and effectiveness of entropy curve control when compared to model scaling alone.

Figure 3: Entrocraft enables smaller models (4B) to surpass larger baselines (8B) and ensures pass@K improves with K, contrasting with baseline entropy collapse.

On math reasoning datasets, Entrocraft improves pass@32 by up to 50% over baseline RL, with mean@32 (expected accuracy) and output diversity consistently raised across all tested tasks and models. Entrocraft also sustains training gains for up to $4\times$ longer than standard policy-gradient approaches before any sign of saturation is evident.

Long-term Training Dynamics and Entropy Schedules

The study systematically explores various entropy annealing schedules:

• Fixed entropy targets quickly become unstable in long-term RL, with samples becoming severely unbalanced, leading to performance fluctuations or sudden collapse.
• Linearly or cosinely decaying entropy schedules enable stable long-horizon learning, with linearly decaying schedules yielding the best and most robust outcomes.

Figure 4: Long-term training stability comparisons: fixed entropy targets (blue) become unstable; linear and cosine decay are robust, and linear decay achieves slightly superior performance.

Figure 5: Entrocraft's precise entropy control prevents saturation, maintaining continued performance improvement over standard GRPO in long-term training.

Figure 6: Compared to baselines, Entrocraft tracks user-specified entropy curves with high fidelity, whereas other methods show instability or insufficient responsiveness.

These findings demonstrate that the optimal entropy trajectory for RL in LLMs is not static, but dynamically annealed: starting at a high value (for exploration) and decaying towards a moderate regime as the policy converges.

Failure Modes of Over-Preserved Entropy

Maintaining excessive entropy can be deleterious. Ablations reveal that fixing entropy at too high a value induces severe instability, increased gradient variance, and ultimately, collapse in downstream task performance—even as training continues.

Figure 7: Overly high entropy targets introduce instability in RL training, making empirical performance fragile to perturbations.

Practical and Theoretical Implications

Entrocraft’s framework formalizes entropy as a directly tunable hyperparameter for LLM RL, fundamentally changing the role of entropy from a passive diagnostic to an optimization lever akin to learning rates or batch sizes. This not only addresses traditional performance saturation and instability problems but also enables systematic ablation and benchmarking of entropy schedules—a previously unattainable degree of control.

Practically, Entrocraft is agnostic to the underlying advantage estimator and policy architecture, making it compatible as a drop-in module with state-of-the-art RL algorithms for LLMs. Empirical gains are apparent on math reasoning datasets, with extensions to other verifiable reward settings such as code generation and multi-turn dialogue RL anticipated. Limitations do exist in regimes such as multi-turn RL or MoE models, where solution sparsity further exacerbates entropy instability, highlighting clear directions for future work.

Conclusion

This work provides rigorous theoretical and practical justification for entropy-guided rejection sampling as a precise, responsive entropy-control tool in LLM RL. The Entrocraft framework overcomes performance saturation induced by entropy collapse, unlocks stable long-horizon improvement, and enables the direct crafting of entropy schedules. This paradigm recasts entropy from an observational statistic to an actionable training hyperparameter, offering a foundation for improved exploration-exploitation balance and scalable RL for next-generation LLMs.