Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of LLMs, particularly on mathematics and programming tasks. Similar to how traditional RL helps agents explore and learn new strategies, RLVR is believed to enable LLMs to continuously self-improve, thus acquiring novel reasoning abilities beyond those of the corresponding base models. In this study we critically examine the current state of RLVR by systematically probing the reasoning capability boundaries of RLVR-trained LLMs across various model families, RL algorithms, and math, coding, and visual reasoning benchmarks, using pass@k at large k values as the evaluation metric. Surprisingly, we find that the current training setup does not elicit fundamentally new reasoning patterns. While RLVR-trained models outperform their base models at small k (e.g., k = 1), the base models achieve a higher pass@k score when k is large. Coverage and perplexity analyses show that the observed reasoning abilities originate from and are bounded by the base model. Treating the base model as an upper bound, our quantitative analysis shows that six popular RLVR algorithms perform similarly and remain far from optimal in leveraging the potential of the base model. By contrast, we find that distillation can introduce new reasoning patterns from the teacher and genuinely expand the model's reasoning capabilities. Overall, our findings suggest that current RLVR methods have not yet realized the potential of RL to elicit truly novel reasoning abilities in LLMs. This highlights the need for improved RL paradigms, such as continual scaling and multi-turn agent-environment interaction, to unlock this potential.

• The paper shows that RLVR biases the output sampling process rather than generating fundamentally new reasoning patterns in LLMs.
• Methodology using the pass@k metric across math, code, and visual tasks reveals that base models catch up as more attempts are allowed.
• Results indicate that distillation can more effectively expand reasoning boundaries compared to the limited gains offered by RLVR.

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Introduction

The paper examines the use of Reinforcement Learning with Verifiable Rewards (RLVR) to enhance reasoning capabilities in LLMs, particularly in mathematics and programming domains. Despite RLVR's empirical success in improving these capabilities, its actual impact on incentivizing novel reasoning patterns beyond what base models can achieve has not been critically evaluated. The paper questions whether RLVR indeed leads to the development of new reasoning abilities or merely biases the sampling process to yield correct answers more efficiently.

Reinforcement Learning with Verifiable Rewards (RLVR)

RLVR applies reinforcement learning to LLMs to enhance reasoning efficiency using automatically computable rewards. These rewards are determined by whether the model's output aligns with true solutions in tasks like mathematics or code correctness. The assumption is that RLVR can lead to self-improvement of LLMs, bringing in advanced reasoning behaviors absent in base models.

Figure 1: The effect of RLVR on LLM's reasoning ability. All reasoning paths in the RLVR model are already present in the base model.

Methodology and Metrics

Pass@$k$ Metric

The researchers employed the pass@$k$ metric, which accounts for the number of correct samples generated out of $k$ attempts. This metric evaluates the boundary of reasoning capabilities of both base and RL-trained models, addressing whether RLVR achieves fundamentally new reasoning patterns.

Experimental Analysis

RLVR in Mathematical Reasoning

Using various LLM families and benchmarks such as GSM8K and MATH500, RLVR-trained models displayed higher pass@1 rates compared to base models. However, as $k$ increased, base models consistently matched or surpassed RLVR models.

Figure 2: Pass@k curves of base models and their zero-RL-trained counterparts across multiple mathematical benchmarks.

RLVR in Code Generation and Visual Reasoning

Similar findings were observed in code generation and visual reasoning tasks. Base models eventually surpassed RLVR models in pass@$k$ scores at larger values of $k$, indicating broader reasoning coverage in base models.

Figure 3: Pass@$k$ curves of base models and zero-RL counterparts. (Left) Code Generation. (Right) Visual Reasoning.

Root Cause Analysis

The findings showed that RLVR does not enhance reasoning capabilities beyond the boundaries established by base models. Perplexity analyses indicated that paths exploited by RLVR are already present in base models' distributions, suggesting that RLVR primarily biases output distribution rather than expanding reasoning ability.

Figure 4: Perplexity distribution of responses from different sources, evaluated by the base and RL models.

Distillation as an Alternative

Distillation, in contrast to RLVR, was found to genuinely expand reasoning boundaries by introducing new knowledge into the model. This suggests that distillation might be a more effective method than RLVR for enhancing reasoning capabilities in LLMs.

Conclusion

The research concludes that RLVR, in its current form, does not contribute fundamentally new reasoning abilities to LLMs but merely enhances sampling efficiency within the reasoning scope of base models. This highlights the necessity for a revised understanding of RLVR's role and suggests a need for more advanced paradigms to exceed the reasoning capacities of LLM base models. Future research might consider integrating alternatives like distillation or novel RL strategies that overcome the limitations identified in this study.