How Far Can Unsupervised RLVR Scale LLM Training?

Abstract: Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.

• The paper demonstrates that intrinsic RLVR methods induce a 'rich-get-richer' effect, amplifying the model’s initial biases and leading to eventual collapse.
• The analysis reveals a rise-then-fall pattern where early improvements in model confidence give way to reward hacking and performance degradation.
• The study introduces the Model Collapse Step metric, offering a practical tool for predicting RL trainability and underscoring the need for scalable external rewards.

How Far Can Unsupervised RLVR Scale LLM Training?—An Analytical Essay

Introduction

The work "How Far Can Unsupervised RLVR Scale LLM Training?" (2603.08660) systematically investigates the prospects and intrinsic limitations of Unsupervised Reinforcement Learning with Verifiable Rewards (URLVR) for LLM training. The authors address a significant challenge as LLMs approach and even exceed human expertise in specialized domains: scalable reward signals without reliance on laborious or often unavailable ground-truth labels. The central focus is to establish whether intrinsic URLVR methods—those using only model-internal signals for reward—can reliably scale LLM training, to what extent they amplify or degrade capabilities, and how alternative external reward schemes may overcome current impediments.

Figure 1: High-level taxonomy of unsupervised RLVR methods and the empirical findings structure the investigation.

Taxonomy of Unsupervised RLVR

The core taxonomy distinguishes between intrinsic rewards, which exploit the model’s own confidence or internal statistics, and external rewards, which ground supervision in signals independent of the current policy. Intrinsic methods are further sub-divided into:

• Certainty-based methods: e.g., Self-Certainty (KL to uniform), entropy minimization, sequence probability, and related metrics reflecting sharpness in predictive distributions.
• Ensemble-based methods: e.g., majority voting across rollouts, semantic consistency, and answer-space aggregation.

These approaches have become increasingly influential in recent open-ended benchmark settings, with practical implementations seen in TTRL, EM-RL, RENT, RLSC, and more.

In contrast, external reward methods use signals derived from the structure or verifiability of unlabeled corpora (e.g., next-token prediction in pretraining data) or leverage computational asymmetries (e.g., generation-verification asymmetry in program synthesis or mathematical proof verification), which are decoupled from model priors.

Theoretical Analysis: Sharpening Mechanism and its Boundaries

The central theoretical result is a convergence theorem for all intrinsic URLVR methods: optimization with intrinsic rewards geometrically concentrates probability on answer(s) already favored by the current policy prior—termed the "sharpening" effect. If the model’s initial predictions are correct, training amplifies those; if incorrect, it amplifies the error.

This "rich-get-richer" dynamic is universal across method families:

• Certainty-based methods reward peaked distributions, accelerating concentration regardless of alignment with correctness.
• Ensemble methods using voting reward reinforce whichever semantic answer dominates the initial rollouts.
• The process is formalized through analysis of the KL-regularized RL objective and closed-form policy updates, with geometric convergence to deterministic policies on the majority solution.

Assumptions for benign convergence (such as stability of the empirical majority and effective policy improvement at every step) are empirically validated for moderate rollout counts.

Figure 2: Training with intrinsic rewards exhibits an initial phase where proxy rewards rise and validation accuracy increases, but later, validation accuracy collapses while proxy reward remains high, highlighting divergence between confidence and correctness.

Empirical Findings: Failure Modes and Limits

Dynamics: Rise-then-Fall Pattern

All intrinsic URLVR methods follow a rise-then-fall pattern: early in training, performance and intrinsic rewards improve as model confidence aligns with correctness. However, eventually, the optimization overshoots, leading to reward hacking and model collapse. Importantly, collapse timing is a function of the model’s prior and not a hyperparameter artifact. Hyperparameter sweeps over temperature, mini-batch, or KL regularization can only delay, not prevent, collapse.

Figure 3: Different intrinsic reward methods fail in distinct ways: Self-Certainty and Majority Voting degrade slowly; Probability-based rewards induce brevity bias and length collapse; entropy-based metrics drive pathological repetition.

Method-specific Failure Patterns

• Self-Certainty and Majority Voting degrade gradually but maintain some label accuracy and do not collapse immediately.
• Trajectory-level and token-level entropy minimization can induce repetition artifacts.
• Probability-based approaches incentivize overly short outputs due to multiplicative effects over tokens.

  Figure 4: Training label accuracy reveals how quickly models overfit pseudo-rewards and lose alignment with ground truth.

Dataset Size and Localized Overfitting

Collapse is mitigated—sometimes altogether avoided—when the training dataset is small (<<500 samples). In such cases, intrinsic reward maximization induces only local overfitting, avoiding global policy shift and catastrophic forgetting. This is quantitatively evident in greatly reduced KL divergence from the reference policy and correlates with previous theoretical work on localized parameter shifts in hyperfitting.

Figure 5: Larger datasets induce global reward hacking, while small-scale settings (≤128) preserve stability and performance.

Safe Applications: Test-Time Training and Model Prior

Training on small, domain-specific datasets at test time remains a practical and beneficial application of intrinsic URLVR. Even if the initial pseudo-labels are largely wrong, performance gains can still accrue on out-of-distribution examples due to cross-problem generalization as long as the prior maintains some confidence/correctness alignment in new domains.

Model Collapse Step: A Practical Prior Indicator

The authors introduce the Model Collapse Step: the RL training step at which accuracy (measured against pseudo-labels or ground truth) collapses below a rigorous threshold. Empirically, this step strongly predicts RL trainability (i.e., which base model will benefit most from downstream RL), correlating with but requiring no access to ground-truth labels. Model Collapse Step outperforms static $pass@k$ metrics, is robust in multiple-choice and generative settings, and enables rapid, label-free prior selection for RL runs.

Figure 6: Later collapse steps correlate with higher RL trainability and greater correct reward gains, outperforming static $pass@k$ statistics.

Additionally, tuning hyperparameters such as mini-batch and rollout number can accelerate diagnostic measurement without altering model ranking.

Figure 7: Increasing rollout count and adopting aggressive batch sizes speed up diagnostic collapse while maintaining ranking fidelity.

External Rewards: Towards Scalable URLVR

External reward methods sidestep intrinsic limitations by generating reward signals from verification processes or unlabeled data structure:

• Unlabeled data-based supervision transforms text completions into reward-bearing tasks (as in RPT, RLPT, TPT).
• Generation-verification asymmetry is exploited in settings where solution production is hard but verification is cheap—math, code, automated proof, games.

These signals scale with available data or computational resources, are not limited by the model’s pre-existing knowledge, and exhibit no rise-then-fall collapse.

Figure 8: Self-Verification (external reward) demonstrates stable, rising validation accuracy across training, contrasting collapse in entropy intrinsic methods.

Instruction alignment improves not only absolute performance but also robustness to prompt design in self-verification settings.

Figure 9: Instruction-aligned models demonstrate higher accuracy and lower sensitivity to verification prompt changes, essential for practical external reward application.

Implications and Future Directions

The fundamental limitation established in this work is that any intrinsic reward for unsupervised RLVR ultimately operates inside the knowledge convex hull of the initialization policy. These mechanisms are strongly beneficial for eliciting and stabilizing existing knowledge (especially in few-shot or test-time adaptation), but cannot create new reasoning ability outside the immediate prior. Large-scale applications require scalable, independent signals—i.e., external rewards.

The implications are multifold:

• For practitioners, model selection for RL runs can use Model Collapse Step with far less computation and annotation cost.
• For test-time tasks, intrinsic URLVR remains safe and beneficial given careful control of dataset size.
• For progress toward superhuman reasoning, infrastructures for scalable unlabeled verification, computational feedback, and formalization of generation-verification asymmetry must be systematized.

Future developments may hybridize intrinsic and external methods, improve reward robustness via dynamic anchoring, or extend computational verification into less-structured domains. How to automate curriculum and verification signal design to maximize discoverability in unaligned or adversarial task settings remains an open theoretical and systems question.

Conclusion

This work rigorously delimits the scalability of intrinsic URLVR for LLM training. All intrinsic approaches, despite design variety, converge to deterministic policies amplifying the model’s prior—enabling efficient exploitation, but not exploration, of knowledge. Collapse is a necessary, not incidental, byproduct and can be diagnosed efficiently via Model Collapse Step. For true capability scaling, the pathway lies through external reward design, leveraging scalable data or cheap verification procedures, decoupled from model bias. This framework provides practical guidance for RLVR application and a research roadmap for the development of self-improving AI systems beyond current limitations.