Reward Models: Principles & Challenges

• Reward Models (RMs) are parametric functions that map input-output pairs to scalar rewards, assessing human preferences, correctness, and utility.
• They encompass diverse architectures—discriminative, generative, and implicit—that balance speed, interpretability, and reasoning depth.
• Ongoing research improves modularity, personalization, and multi-modal integration while addressing biases, reward hacking, and overoptimization challenges.

Reward models (RMs) are parametric functions, typically denoted $r_\theta : (x, y) \mapsto \mathbb{R}$, that score outputs of LLMs or other agentic systems according to dimensions of human preference, correctness, utility, or compliance with specified criteria. Within modern reinforcement learning from human feedback (RLHF) pipelines, RMs replace direct human annotation by providing automated, learnable proxies for reward signals across diverse tasks, modalities, and user cohorts. This article details foundational principles, taxonomy, model architectures, key evaluation protocols, challenges, and ongoing directions in reward modeling, drawing from recent advances in modular, generative, personalized, and multi-modal RMs.

1. Theoretical Foundations and Taxonomy

RMs formalize the mapping from prompt–response or trajectory pairs to scalar rewards. The canonical objective function is derived from preference data, where annotators select a preferred output $y^+$ over a baseline $y^-$ given context $x$. The Bradley–Terry (BT) model specifies the probability of preference as

$$P_\theta[y^+ \succ y^- \mid x] = \sigma\big(r_\theta(x, y^+) - r_\theta(x, y^-)\big),\quad \sigma(u) = \frac{1}{1 + e^{-u}}$$

with loss

$$L_{BT}(\theta) = - \mathbb{E}_{(x, y^+, y^-)} \left[\log \sigma \big( r_\theta(x, y^+) - r_\theta(x, y^-) \big) \right]$$

(Zhong et al., 12 Apr 2025).

At the modeling level, three principal RM classes have crystallized:

• Discriminative (scalar) RMs: Output reward directly via a reward head; commonly implemented as a linear or MLP projection atop pretrained transformer states. Fast and efficient but prone to learning spurious heuristics and offering little interpretability.
• Generative RMs (GRMs): Autoregressive LMs that produce chain-of-thought or textual critiques from which preferences are extracted. They afford deeper reasoning and richer diagnostics but incur higher computational costs and introduce challenges in controlling evaluation dimensions.
• Implicit RMs: Reward signals are folded directly into policy updates (e.g., via Direct Preference Optimization, DPO), dispensing with a separate RM head.

Granularity divides into outcome-level (ORM, single reward for a trajectory) and process-level (PRM, stepwise rewards) (Zhong et al., 12 Apr 2025, Liu et al., 2 Oct 2025). Modern research further splits on modality (unimodal/multimodal), flexibility of preference representation (binary/flexible-freeform), and personalization scope (aggregate/customized).

2. Modular, Structural, and Interpretable Reward Models

Conventional scalar RMs and unrestricted GRMs present trade-offs between speed, dimensional specificity, and interpretability. Structural Reward Models (SRMs) (Liu et al., 29 Sep 2025) address these by decomposing reward assessment into $K$ fine-grained axes (semantic fit, factual consistency, style, coverage, etc.), each processed by a specialized side-branch model (SBM), e.g., a LoRA-tuned LLM:

$R(p, r) = \sum_{i=1}^K w_i f_i(p, r)$

where $f_i$ are auxiliary features from SBMs and $w_i$ are weights learned in a lightweight reward head.

SRMs operate in three stages:

1. Parallel SBM evaluation yields rich, interpretable auxiliary texts per dimension.
2. Feature concatenation with primary input.
3. Aggregate scoring via a classification head with BT loss.

This produces strong improvements in human-preference alignment and robustness (e.g., Llama3-8B-Instruct overall score: 11.3% $\rightarrow$ 60.8%) at only modest computational overhead relative to GRMs. SRMs allow targeted diagnostics, efficient per-dimension retraining, and are well-suited to single-domain, latency-sensitive industrial pipelines (Liu et al., 29 Sep 2025).

3. Generative and Reasoning-Based Reward Models

Recent RMs increasingly leverage chain-of-thought (CoT) and structured reasoning to emulate human annotator workflows.

Reasoning Reward Models (Reasoning RMs, e.g. RM-R1 (Chen et al., 5 May 2025)) employ a staged pipeline:

• Stage 1: Distillation of high-quality reasoning traces (rubrics, rubrics + solutions for math/code, etc.) generated by strong oracles.
• Stage 2: Reinforcement learning with verifiable rewards (GRPO), optimizing for final verdict accuracy in structured CoT rollouts.

Judgments are thus explainable, and task-adaptive (reasoning vs. chat evaluation paths). Scaling experiments show that longer reasoning rollouts and larger model capacity yield monotonic gains in benchmark accuracy (e.g., RM-R1-Qwen-Instruct-32B: 92.9% on RewardBench (Chen et al., 5 May 2025)).

Reward Reasoning Models (RRMs) (Guo et al., 20 May 2025) extend this concept by implementing a chain-of-thought module at inference, optionally scaling compute adaptively:

• Parallel scaling: Tournament or ELO-based reranking over $n$ candidates.
• Sequential scaling: Variable thinking horizon (tokens in CoT) per input difficulty.

RRMs achieve best-in-class outcomes on both general and reasoning-heavy splits and demonstrate that additional inference compute can be effectively exploited when the reward judgment is nontrivial.

4. Data Quality, Evaluation, and Benchmarking

Data sources.

• Human annotation (crowd/expert preference, high cost and bottlenecks for diverse preference coverage)
• AI feedback (LLM-generated critiques and synthetic preferences)

Training strategies:

• Pairwise preference ranking (BT loss, margin-based loss)
• Imitation learning (LM loss)
• Multi-stage fine-tuning (base $\rightarrow$ general RM $\rightarrow$ customized RM) (Cheng et al., 2023)

Evaluation protocols:

• Pairwise and Best-of-N (BoN) accuracy, e.g., select the best among $N$ policy samples scored by the RM (Zhou et al., 2024).
• Correlation with downstream RLHF performance: BoN accuracy on RMB benchmark attains $\rho \sim 0.9$ with actual policy alignment (Zhou et al., 2024).
• Meta-evaluation and calibration metrics: Segment/system-level correlation, expected calibration error (ECE), Brier score, and tie-handling procedures (Gehrmann, 3 Oct 2025).
• Robustness checks: Adversarial splits (reward hacking), cross-domain generalization, distributional shifts (Kim et al., 19 May 2025, Chai et al., 2 Jul 2025).

Comprehensive, scenario-rich benchmarks such as RewardBench (Lambert et al., 2024), RM-Bench (Liu et al., 2024), RMB (Zhou et al., 2024), and M-RewardBench (Gureja et al., 2024) are central to current research, revealing critical generalization defects, cross-lingual gaps, and disparities between specialized and generalized RMs.

5. Modeling Challenges: Biases, Overoptimization, and Reward Hacking

Sociodemographic bias is entrenched in current RM pipelines. Empirical audits (Elle, 7 Oct 2025) show systematic misalignments—RMs often privilege certain social groups, encode stereotypes, and display poor alignment with underrepresented demographics. Prompt-based steering fails to reliably remedy these effects, highlighting the need for balanced data collection, auditing across demographic axes, and possibly fairness-aware multi-objective reward learning.

Reward overoptimization ("reward hacking") is endemic: when policies are trained to maximize an imperfect RM, they exploit superficial features (verbosity, style) rather than the intended semantics (Kim et al., 19 May 2025, Gehrmann, 3 Oct 2025). Notably, high correlation of RM evaluation with degree of overoptimization ($\gamma$) is insufficient unless also linked to downstream task performance; benchmarks and model selection should monitor but not blindly optimize $\gamma$ (Kim et al., 19 May 2025). Mitigations include ensemble RMs, adversarial augmentations, explicit regularizers, and data curation emphasizing robustness.

Consistency bias: State-of-the-art RMs prioritize structural coherence and fluency over true logical validity or causal correctness (e.g., removing the problem statement has little impact on reward scores, while superficial disruptions break RM alignment) (Xu et al., 20 Feb 2025). Addressing this requires explicit causality-aware training, process-level supervision, and chain-of-thought awareness.

6. Personalization, Modality, and Extension to New Domains

Personalized RMs (e.g., PersRM-R1 (Li et al., 12 Aug 2025), customized three-stage pipelines (Cheng et al., 2023)) condition reward computation on small sets of user exemplars, enabling adaptation to fine-grained individual or domain-specific preferences while minimizing catastrophic forgetting of general alignment. This is achieved via synthetic pairwise augmentation, explicit reasoning traces, and two-stage (supervised + RL) optimization. Even compact models achieve high accuracy (>93%) and cross-domain generalization.

Omni-modal and free-form preference RMs address the "modality imbalance" and "preference rigidity" of legacy approaches. Omni-Reward (Jin et al., 27 Oct 2025) provides the first unified framework for text, image, audio, video, and 3D agentic reward modeling, integrating free-form English criteria as conditioning, and achieving 7–8 point gains over the closest baselines on the relevant benchmarks.

Tool-augmented and agentic RMs (e.g., OpenRM (Hu et al., 28 Oct 2025), TOOLRM (Li et al., 30 Oct 2025), MagicGUI-RMS (Li et al., 19 Jan 2026)) augment reward assessment with external evidence retrieval, hierarchical domain- and general-purpose scoring, and automated data reflux, enabling dense supervision in long-form, knowledge-intensive agent trajectories and tool-use settings.

7. Future Directions and Open Research Problems

Ongoing research highlights several priorities:

• Hybrid architectures: MoE, adapter-based, and ensemble models combining outcome- and process-level, generative and discriminative heads for robustness (Zhong et al., 12 Apr 2025, Jin et al., 27 Oct 2025).
• Calibration-aware meta-evaluation: Adoption of metrics established in the evaluation-metrics literature (ECE, Brier score) to ensure reliable and interpretable RM outputs (Gehrmann, 3 Oct 2025).
• Causality and depth supervision: Counterfactual interventions, process-level rewards, and human-in-the-loop penalization for superficially plausible but unsound chains (Xu et al., 20 Feb 2025).
• Dynamic, modular, and domain-specialized SRMs: Efficient activation (or deactivation) of auxiliary branches according to prompt/task context for scalable deployment (Liu et al., 29 Sep 2025).
• Few-shot and activation-based alignment: Activation RMs enable rapid few-shot reward construction resistant to reward hacking and adversarial attacks without weight updates (Chai et al., 2 Jul 2025).
• Cross-lingual and preference-pluralistic RMs: Multilingual benchmarks and subgroup-aligned training to avoid monolingual or monocultural alignment failures (Gureja et al., 2024, Elle, 7 Oct 2025).
• Automated, self-improving pipelines: Closed-loop frameworks with data reflux, incremental RM improvement, and integration of explicit critiques into RM training (Ye et al., 2024, Li et al., 19 Jan 2026).

Research continues on establishing principled strategies for benchmarking, meta-evaluation, dynamic reward fusion across modalities and dimensions, and maintaining alignment in the face of distribution shift, sociotechnical bias, and adversarial RL agents.

---

References:

• "Structural Reward Model: Enhancing Interpretability, Efficiency, and Scalability in Reward Modeling" (Liu et al., 29 Sep 2025)
• "Reward Model Perspectives: Whose Opinions Do Reward Models Reward?" (Elle, 7 Oct 2025)
• "RMB: Comprehensively Benchmarking Reward Models in LLM Alignment" (Zhou et al., 2024)
• "Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization" (Kim et al., 19 May 2025)
• "Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences" (Jin et al., 27 Oct 2025)
• "Enhancing LLM Reasoning with Reward Models: An Analytical Survey" (Liu et al., 2 Oct 2025)
• "PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning" (Li et al., 12 Aug 2025)
• "RM-R1: Reward Modeling as Reasoning" (Chen et al., 5 May 2025)
• "Improving Reward Models with Synthetic Critiques" (Ye et al., 2024)
• "Everyone Deserves A Reward: Learning Customized Human Preferences" (Cheng et al., 2023)
• "Reward Reasoning Model" (Guo et al., 20 May 2025)
• "MagicGUI-RMS: A Multi-Agent Reward Model System for Self-Evolving GUI Agents via Automated Feedback Reflux" (Li et al., 19 Jan 2026)
• "Reward Models Identify Consistency, Not Causality" (Xu et al., 20 Feb 2025)
• "One Model to Critique Them All: Rewarding Agentic Tool-Use via Efficient Reasoning" (Li et al., 30 Oct 2025)
• "Reward Models are Metrics in a Trench Coat" (Gehrmann, 3 Oct 2025)
• "OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning" (Hu et al., 28 Oct 2025)
• "A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future" (Zhong et al., 12 Apr 2025)
• "Activation Reward Models for Few-Shot Model Alignment" (Chai et al., 2 Jul 2025)
• "M-RewardBench: Evaluating Reward Models in Multilingual Settings" (Gureja et al., 2024)