Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains

Abstract: Reinforcement learning with verifiable rewards (RLVR) has demonstrated significant success in enhancing mathematical reasoning and coding performance of LLMs, especially when structured reference answers are accessible for verification. However, its extension to broader, less structured domains remains unexplored. In this work, we investigate the effectiveness and scalability of RLVR across diverse real-world domains including medicine, chemistry, psychology, economics, and education, where structured reference answers are typically unavailable. We reveal that binary verification judgments on broad-domain tasks exhibit high consistency across various LLMs provided expert-written reference answers exist. Motivated by this finding, we utilize a generative scoring technique that yields soft, model-based reward signals to overcome limitations posed by binary verifications, especially in free-form, unstructured answer scenarios. We further demonstrate the feasibility of training cross-domain generative reward models using relatively small (7B) LLMs without the need for extensive domain-specific annotation. Through comprehensive experiments, our RLVR framework establishes clear performance gains, significantly outperforming state-of-the-art open-source aligned models such as Qwen2.5-72B and DeepSeek-R1-Distill-Qwen-32B across domains in free-form settings. Our approach notably enhances the robustness, flexibility, and scalability of RLVR, representing a substantial step towards practical reinforcement learning applications in complex, noisy-label scenarios.

• The paper introduces RLVR, a novel method that assigns verifiable rewards to LLM responses to improve reasoning across diverse domains.
• It employs a model-based soft scoring system with z-score normalization to stabilize training and outperform binary reward schemes.
• Experimental results show that a 7B parameter reward model trained on distilled data achieves comparable performance to larger models.

Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains

Introduction

The paper "Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains" introduces Reinforcement Learning with Verifiable Rewards (RLVR) as a method to improve the reasoning capabilities of LLMs across diverse domains. It explores using objective reference answers to verify model responses, challenging the necessity of extensive annotations for reward models. The incorporation of model-based soft scoring is proposed to enhance flexibility in handling unstructured reference answers.

Figure 1: Overview paradigm of RLVR with our cross-domain verifier.

Methodology

Reward Estimation

The RLVR method assigns verifiable reward signals to LLM responses based on comparison with reference answers. A binary reward function is initially used, where the signal is either 0 or 1, depending on the model's response matching the ground truth. A soft reward function, leveraging the probability of tokens representing the verifier's judgment, is proposed for more nuanced scoring. Normalization of rewards using z-score further stabilizes training gradients, ensuring that favorable samples exceed average performance.

Cross-Domain Reward Model

A significant contribution is the distilled generative reward model, which serves as a cross-domain verifier, trained without domain-specific annotations. This model is developed through supervised learning on data collected during RL exploration sessions. The adaptive nature of responses, even with formatting noise, is posited to improve reward model robustness.

Experimental Results

The authors employ various RL algorithms, including REINFORCE, RLOO, and REINFORCE++, to test their method across domains such as mathematics and multi-subject tasks. They demonstrate RLVR's effectiveness compared to supervised fine-tuning (SFT) and rule-based rewards.

Findings:

• Model-based soft rewards consistently outperform binary rewards, particularly in multi-domain settings where answer diversity complicates exact matches.
• A 7B parameter reward model, optimized against distilled data from a larger model (Qwen2.5-72B-Instruct), achieves comparable performance in reasoning tasks.
• Scaling experiments reveal the superior adaptability of model-based rewards to increasing data size compared to rule-based rewards.

  Figure 2: Agreement between GPT-4o and Majority Vote with m Graders, measured by Cohen's Kappa.

Discussion

The paper underscores the potential of RLVR to redefine how LLMs handle reasoning tasks beyond traditional domains. The reduced dependency on domain-specific annotations and the ability to train an effective verifier using a smaller LLM highlight scalability and efficiency. The exploration of model-based rewards contrasts the rigidity of rule-based systems, opening pathways for more adaptable learning methodologies. Future work could explore the integration of process reward modeling to structure intermediate verifications more proficiently.

Conclusion

The study successfully expands the scope of RLVR to diverse domains, proving its versatility and effectiveness in handling unstructured reference answers. The insights gained can lead to enhanced scalability and robustness in real-world applications, minimizing the limitations associated with traditional rule-based rewards systems.