IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL

Abstract: While scaling laws guide compute allocation for LLM pre-training, analogous prescriptions for reinforcement learning (RL) post-training of LLMs remain poorly understood. We study the compute-optimal allocation of sampling compute for on-policy RL methods in LLMs, framing scaling as a compute-constrained optimization over three resources: parallel rollouts per problem, number of problems per batch, and number of update steps. We find that the compute-optimal number of parallel rollouts per problem increases predictably with compute budget and then saturates. This trend holds across both easy and hard problems, though driven by different mechanisms: solution sharpening on easy problems and coverage expansion on hard problems. We further show that increasing the number of parallel rollouts mitigates interference across problems, while the number of problems per batch primarily affects training stability and can be chosen within a broad range. Validated across base models and data distributions, our results recast RL scaling laws as prescriptive allocation rules and provide practical guidance for compute-efficient LLM RL post-training.

• The paper introduces an empirical framework establishing scaling laws for allocating parallel rollouts (n) optimally within a fixed sampling compute budget for LLM RL.
• It demonstrates that increasing parallel rollouts enhances solution robustness on easy tasks and trajectory discovery on challenging prompts until a context-dependent saturation point is reached.
• The study provides actionable guidelines by analyzing the interplay of compute allocation parameters to mitigate overfitting and optimize performance in LLM RL.

IsoCompute Playbook: Optimal Sampling Compute Allocation for LLM Reinforcement Learning

Introduction

Scaling LLM reinforcement learning (RL) to maximize downstream performance under sampling compute constraints is nontrivial due to intricate couplings between data collection, model optimization, and problem distribution characteristics. Unlike the pre-training regime where scaling laws are well-studied, RL post-training for LLMs lacks prescriptive, empirical rules for compute allocation, particularly for on-policy algorithms involving parallel rollouts, diverse prompt batches, and multi-step optimization. The "IsoCompute Playbook: Optimally Scaling Sampling Compute for LLM RL" (2603.12151) provides a comprehensive empirical framework and actionable scaling laws for allocating sampling compute along key axes in LLM RL.

Figure 1: Compute-optimal sampling for LLM RL—parallel rollouts per problem ($n$), problems per batch ($B_{\text{p}}$), and sequential iterations ($M$); highlights main empirical findings.

Problem Setup and Methodological Framework

The analysis considers a fixed sampling compute budget, $C$, factorized as $C = n \cdot B_{\text{p}} \cdot M$, where $n$ is parallel rollouts per problem (exploration width), $B_{\text{p}}$ is the number of unique problems per batch (diversity per update), and $M$ is the number of sequential gradient steps (optimization duration). The study encompasses extensive sweeps ($120,000+$ H200 GPU hours) over multiple architectures (Qwen2.5-7B-Instruct, Qwen3-4B-Instruct, Llama-3.1-8B-Instruct), problem difficulty regimes, and various training configurations.

A critical precursor to scaling law discovery is a healthy RL recipe. RL post-training is highly sensitive to the regularization regime; instabilities such as entropy collapse or explosion, particularly on hard or heterogeneous prompts, demand a tailored mix of entropy/KL regularization. For easy prompts, KL and entropy regularization prevent premature collapse, while for hard prompts, they are omitted to avoid destabilizing optimization.

Figure 2: Square-root learning-rate scaling ($\eta \propto \sqrt B$) ensures stable, high-throughput RL optimization at scale.

Compute-Optimal Scaling Laws: Empirical Findings

Parallel Rollouts ($n$) vs Sequential Iterations ($M$)

The core empirical law is that the compute-optimal number of parallel rollouts per problem, $n^*(C)$, increases with available compute budget—initially rapidly then saturating according to a sigmoid in log-compute—unlike intuition from tabular RL where more sequential updates are always optimal.

Figure 3: Generalization of $n$ scaling trends to other model families and settings; optimal $n$ grows with compute but saturates at a context-dependent threshold.

This effect is robust across easy and hard problems, but driven by different mechanisms. For easy problems, increasing $n$ primarily sharpens the solution—improving worst-case solution robustness (worst@k); for hard problems, larger $n$ is critical for expanding coverage, detecting rare successful trajectories, and boosting best@k. Small $n$ can induce optimization-induced interference—some prompts remain unsolved despite being tractable.

Figure 4: Record-breaking "frontier" points extraction: fitting monotonic scaling laws for optimal $n$ as a function of compute budget.

Problems per Batch ($B_{\text{p}}$): Role and Effect

For fixed hardware (total batch size $B$), allocating more compute to $n$ (i.e., fewer unique problems per batch, but deeper exploration per problem) is compute-optimal at high $M$. At low $M$, covering more problems per update (higher $B_{\text{p}}$) is preferable to obtain broad training signal.

Figure 5: Compute-optimal frontiers on the Easy set under fixed total batch size for different $B$ values.

On easy problem sets, performance is significantly more sensitive to $n$ than to $B_{\text{p}}$; on hard sets, both parameters influence reward, necessitating more nuanced dynamic allocation. Empirically, stable $B_{\text{p}}$ suffices within a broad range; $n$ primarily governs the compute-optimal allocation, except at extremely high or low values.

Joint Optimization: $n$, $B_{\text{p}}$, $M$

Under unconstrained joint optimization, the previously established law persists: as compute increases, the optimal $n$ grows predictably and then saturates, while $B_{\text{p}}$ can be held nearly constant for most configurations as a stability knob, with $M$ absorbing the remaining allocation.

Figure 6: Compute-optimal parallel rollouts $n^*(C)$ as the governing degree of freedom; joint sweeps show consistent monotonic scaling with compute.

Regime-Specific and Practical Insights

• The saturation point of $n$ is context-dependent: base model capacity, prompt distribution, and dataset size influence when further increasing $n$ ceases to provide returns and may induce overfitting (i.e., over-optimization on small or easy prompt sets).
• On extremely hard problem subsets (e.g., pass@128=0), a coverage–sharpening tradeoff emerges: best@k is maximized at high $n$, but worst@k can degrade if $n$ is oversized, emphasizing metric-sensitive allocation.

  Figure 7: On extremely hard subsets, best@4 continues to benefit from increasing $n$, but worst@4 is maximized at a moderate $n$, revealing a coverage-sharpening trade-off.

• All scaling laws and prescriptions are validated across algorithms (GRPO, PPO, CISPO) and hardware budgets, and trends are invariant to measuring compute by rollouts or tokens.

  Figure 8: Sigmoid-fit for $\log_2 n^*$ is robust whether compute is defined by number of rollouts or tokens.

Analysis: Interference, Overfitting, and Prompt Distribution Dependency

The dominant theoretical departure from classical RL or tabular bandits arises from interference in population learning: multiple prompts can compete for optimization, causing some to remain unsolved unless sufficiently parallel rollouts are allocated. Larger $n$ reduces the variance of gradient updates and ensures more uniform progress across prompts, mitigating "rich-get-richer" dynamics and catastrophic forgetting. Small datasets saturate rapidly—further increasing $n$ quickly induces overfitting detectable by decoupling training and validation performance.

Figure 9: With increased data size, scaling performance improves; with very small data, validation reward degrades for too-large $n$, showing prompt-overfitting.

Finally, the paper's scaling law is robust to varying model families and prompt distributions, but the optimal $n$ is anchored to training set difficulty and model capacity, so practical deployment must calibrate these empirical prescriptions.

Implications and Future Directions

This work recasts RL scaling for LLMs as a compute-constrained resource allocation problem, emphasizing that the optimal axis to scale, the saturation regime, and even the optimal regularization schedule are prompt-distribution dependent. Practically, the primary tuning axis is always parallel rollouts per problem ($n$); other hyperparameters act mainly as secondary stability or throughput parameters when truncated to a broad, stable regime.

The results provide immediately actionable guidelines for practitioners: for any fixed hardware or compute budget, first calibrate $n$ according to the presented scaling law, adjust $B_{\text{p}}$ for stability, and only then tune $M$ for total sequential optimization.

Theoretically, interference and optimization-induced forgetting observed in these large-population RL settings opens questions for future study—e.g., dynamic problem selection, adaptive mixture strategies, and early-stopping protocols to mitigate overfitting and interference. Additionally, tracking sufficient statistics (such as pass@1 histograms) during RL could enable early prediction of learning progress and more accurate compute allocation.

Conclusion

The IsoCompute Playbook establishes principled, empirically derived scaling laws for sampling compute allocation in LLM RL, validated across models, difficulty regimes, and training protocols. Optimal allocation systematically and sigmoidal increases parallel rollouts per problem as compute scales, then saturates at context-dependent thresholds. These results yield a practical, robust workflow for compute-efficient RL post-training, integrate seamlessly into modern LLM pipelines, and motivate deeper theoretical investigation into interference and population learning phenomena in large-batch RL.