Generative Reward Models (GRMs)

• Generative Reward Models (GRMs) are reward frameworks that generate both natural-language rationales and explicit reward signals for enhanced interpretability and policy supervision.
• They employ sequence-generation techniques in pairwise and pointwise settings, enabling scalable inference through ensemble scoring and dynamic computation allocation.
• Empirical results, such as with the IRPO approach, show improved accuracy and computational efficiency, making GRMs effective for RL alignment and transparent decision-making.

Generative Reward Models (GRMs) are a class of reward modeling architectures that leverage the conditional generation capabilities of LLMs or multimodal LLMs to encode, interpret, and refine human preference signals for reinforcement learning, alignment, and evaluation across diverse domains. Unlike scalar reward models, which map an input–output pair to a single score, GRMs generate both natural-language rationales and explicit reward signals, supporting richer interpretability, dynamic scaling, and robust generalization. The GRM framework encompasses a variety of training and inference methodologies, ranging from classical pairwise models to scalable pointwise and intergroup extensions, each optimized for different computational, theoretical, and application requirements.

1. Generative Reward Model Fundamentals

At their core, GRMs treat reward modeling as a sequence-generation task, in which an LLM is prompted with an input (e.g., x: prompt, y: candidate response) and outputs both a critique (chain-of-thought, CoT) and a scalar score s. This score serves as a direct reward in RL-based policy optimization, while the rationale ensures model interpretability and auditability (Song et al., 2 Jan 2026). The GRM's output can be structured as a joint probability

$p_\theta(R, s \mid x, y)$

where $R$ is the reasoning trace and $s$ the scalar score or preference decision. In classification contexts, the GRM may additionally generate an answer indicator ($I \in \{\mathrm{A},\mathrm{B}\}$), modeling pairwise preferences as

$$p_\theta(y_1 \succ y_2 | x) = \pi_\theta(I = A | x, y_1, y_2)$$

(Mahan et al., 2024). GRMs commonly support inference-time scalability: by invoking repeated CoT steps, ensembling via sampling, or parallel generation, additional compute directly increases scoring reliability and discrimination (Song et al., 2 Jan 2026, Liu et al., 3 Apr 2025).

2. Computational Complexity: Pairwise vs. Pointwise and IRPO

Traditional pairwise GRMs suffer from an $O(n^2)$ time complexity when evaluating n candidate responses, as every response pair $(y_i, y_j)$ must be compared to obtain relative scores. This creates a bottleneck in RL settings such as Group Relative Policy Optimization (GRPO), particularly when repeated sampling or CoT reasoning increases cost further (Song et al., 2 Jan 2026). To circumvent this, IRPO (Intergroup Relative Preference Optimization) partitions candidates into "chosen" and "rejected" groups, samples completions within each group, and computes pointwise intergroup rewards using preference-metric rules (e.g., average pairwise sigmoid, Hanley–McNeil AUC) or rule-based interval methods. These metrics reduce evaluation to $O(n)$ complexity and maintain interpretability—each reward is associated with a specific rationale. Empirically, IRPO achieves SOTA performance for pointwise GRMs, with accuracy (voting@8) of 75.1% (+4.6% over the TIR baseline) across benchmarks, and matches pairwise GRM accuracy while reducing reward inference time by up to 4× (Song et al., 2 Jan 2026).

Model Type	Complexity	Interpretability
Pairwise GRM	O(n²)	High (CoT)
IRPO GRM	O(n)	High (CoT/rules)
Scalar RM	O(n)	Low

Rule design is critical: median-, mean-, and interval-based rules produce stable training and accuracy; preference-strength metrics often destabilize intragroup variance.

3. Reward Model Training and Inference Protocols

GRMs are trained using Maximum Likelihood Estimation (MLE) for CoT and reward sequence prediction, or via policy-gradient objectives that use the scalar score as advantage in RL fine-tuning. The reward signal is often defined by classical models (e.g., Bradley–Terry): $$\mathcal{L} = -\sum_{(i,j)} \log \sigma(s_i - s_j), \quad \sigma(z) = 1/(1 + e^{-z})$$ GRPO or PPO-style RL updates are applied, incorporating clipped gradients and KL penalties to regularize divergence from reference policies. In IRPO, reward computation is embedded in a batched RL loop, with groupwise advantage normalization and pointwise GRM scores. Fine-tuning typically iterates over batches of prompts and completions, running a policy update after reward computation (Song et al., 2 Jan 2026).

Unlike scalar RMs, GRMs' sequence-based generation allows extraction of richer, evaluative rationales, and supports scalable parallel inference for more robust policy supervision (Liu et al., 3 Apr 2025). GRMs can be adapted to both pairwise (relative preference) and pointwise (absolute preference) supervision tasks.

4. Interpretability and Inference-Time Scalability

The interpretability of GRMs stems from the explicit chain-of-thought or natural language justification generated for each scoring decision. This rationale permits human inspection and auditing, and can be further analyzed for debugging, adversarial assessment, or downstream selection (Liu et al., 3 Apr 2025, Song et al., 2 Jan 2026). Inference-time scalability is realized by generating multiple CoT rollouts per candidate and aggregating results via voting or meta-model-guided selection, enabling dynamic allocation of compute resources to high-uncertainty or high-impact decisions. This property makes GRMs particularly suited to contexts where increased compute directly benefits reward fidelity, such as test-time best-of-N candidate selection, post-training filtering, and ensemble-based evaluation (Liu et al., 3 Apr 2025).

5. Empirical Performance and Benchmarks

Extensive benchmark testing demonstrates that advanced GRMs—especially IRPO-style pointwise models—achieve parity with or outperform previous pairwise GRMs across domains such as chat, code, math, and safety. In four major benchmarks (PPE Preference & Correctness, RM-Bench, JudgeBench, RewardBench), IRPO achieves average accuracy of 75.1% (+4.6% over TIR pointwise baseline), comparable to leading pairwise models but at a fraction of computational overhead (Song et al., 2 Jan 2026). Ablations reveal that CoT-based rewards outperform pure score-based approaches in post-training and policy update settings; superiority persists in post-training evaluations on tasks like WebInstruct and MMLU-Pro.

6. Limitations and Future Research Trajectories

Current GRM limitations include reliance on rule-based reward thresholds (handcrafted metric/rule definitions), sensitivity to tie rates (~10%), and potentially brittle performance under novel domains or adversarial inputs. The mapping from CoT rationale or intergroup comparison to final reward remains largely static; future directions propose joint learning of reward design functions with policy optimization, Bayesian integration for uncertainty-aware scoring, and exploration of richer group structures beyond two-way partitioning (Song et al., 2 Jan 2026). There is scope for more robust learning of rubric adaptation, dynamic calibration, and hierarchical preference modeling. Integration of uncertainty estimation (e.g., Bayesian approaches) for reward signals merits further study.

7. Significance and Impact of GRMs in RL Alignment

GRMs have fundamentally reshaped reward model design in RLHF and broader alignment pipelines by combining interpretability, scalability, and adaptability. By shifting reward generation from opaque scalar heads to explicit generative reasoning, GRMs enable policies to be fine-tuned not just for preference alignment but also for process transparency and auditability. The IRPO framework demonstrates that computational efficiency need not be traded off against reward fidelity: with pointwise intergroup scoring and reasoned justification, high-performance RL can scale to large candidate pools and complex tasks (Song et al., 2 Jan 2026). These advances inform both practical deployments and theoretical research on robust, transparent preference modeling in high-dimensional action spaces.