RLPR: Extrapolating RLVR to General Domains without Verifiers

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates promising potential in advancing the reasoning capabilities of LLMs. However, its success remains largely confined to mathematical and code domains. This primary limitation stems from the heavy reliance on domain-specific verifiers, which results in prohibitive complexity and limited scalability. To address the challenge, our key observation is that LLM's intrinsic probability of generating a correct free-form answer directly indicates its own evaluation of the reasoning reward (i.e., how well the reasoning process leads to the correct answer). Building on this insight, we propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains. RLPR uses the LLM's own token probability scores for reference answers as the reward signal and maximizes the expected reward during training. We find that addressing the high variance of this noisy probability reward is crucial to make it work, and propose prob-to-reward and stabilizing methods to ensure a precise and stable reward from LLM intrinsic probabilities. Comprehensive experiments in four general-domain benchmarks and three mathematical benchmarks show that RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models. Notably, RLPR outperforms concurrent VeriFree by 7.6 points on TheoremQA and 7.5 points on Minerva, and even surpasses strong verifier-model-dependent approaches General-Reasoner by 1.6 average points across seven benchmarks.

• The paper’s main contribution is introducing a probability-based reward framework that eliminates the need for domain-specific verifiers.
• It leverages intrinsic token probabilities from LLMs to generate stable reward signals, resulting in significant improvements in reasoning tasks.
• The framework incorporates debiasing and filtering techniques to ensure robustness against variations in prompt design while enhancing scalability.

RLPR: Extrapolating RLVR to General Domains without Verifiers

The RLPR framework aims to advance Reinforcement Learning with Verifiable Rewards (RLVR) by removing its dependence on domain-specific verifiers, thereby enhancing reasoning capabilities across general domains. Central to RLPR is utilizing the intrinsic token probabilities from LLMs as a reward signal, which eliminates the complexity and scalability issues tied to traditional verifier-based methods.

RLPR Framework

Motivation and Design

RLVR has traditionally relied on domain-specific verifiers to evaluate the correctness of generated responses, especially in mathematical and coding domains. However, the reliance on verifiers leads to high complexity and limited scalability when expanded to general domains. RLPR proposes replacing these external verifiers with the intrinsic probability scores generated by the LLM itself, leveraging its foundational capabilities for reward generation.

Figure 1: The RLPR framework replaces complex verifier-based rewards with simple probability-based rewards generated by the policy model, enhancing natural language processing capabilities.

Probability Reward Computation

The concept of Probability Reward in RLPR involves calculating rewards based on the LLM's internal evaluation of response quality via token probabilities. Specifically, the framework computes the mean probability of the reference answer's tokens to derive a stable and precise reward signal. This mechanism is more robust than traditional sequence likelihood, as it mitigates the high variance and instability in reward signals caused by minor variations such as synonyms.

Debiasing and Filtering

To address biases in probability-based rewards, the RLPR incorporates a debiasing method that adjusts rewards by comparing the standard decoding of reference answers without reasoning. Additionally, RLPR employs a standard deviation filtering strategy to exclude prompts with low reward variance, adaptive to shifts in reward distribution throughout training.

Experimental Results

Performance

Comprehensive experiments demonstrate RLPR's substantial improvements in reasoning tasks across both mathematical and general domains. For instance, RLPR achieved a notable average accuracy improvement over existing RLVR methods such as General Reasoner and concurrent approaches like VeriFree (Figure 2).

Figure 2: RLPR achieves consistent improvements in both mathematical and general-domain benchmarks, outperforming strong RL methods driven by verifier rewards.

Robustness

Testing across different training prompt templates reveals RLPR's consistent performance and resilience to variations in prompt design, response length, and entropy, exhibiting minimal sensitivity compared to other methods.

Figure 3: RLPR demonstrates robustness across different training prompt templates, maintaining high performance and stable response characteristics.

Analysis of Probability Reward

Quality and Effectiveness

Through token-level probability analysis and quantitative ROC-AUC evaluation, Probability Reward displays superior discrimination capability compared to rule-based verifiers and trained verifier models. The per-token approach yields stable reward signals, correlating highly with response quality across diverse domains.

Enhancing Verifiable Domains

In domains with established verifiers, RLPR further enhances reasoning by combining rule-based and probability rewards, demonstrating improved utilization of mathematical data in tasks such as theorem proving.

Conclusion

RLPR marks a significant step towards scalable domain-general reinforcement learning by utilizing intrinsic LLM capabilities for reward generation, eschewing external verifiers. The framework's innovative approach to reward computing and training stability through debiasing and filtering strategies sets the stage for broader applications, including multimodal understanding. Future exploration will focus on expanding RLPR to larger models and various complex domains.