RoleRM: Reward Modeling for LLM Role-Play

• RoleRM is a specialized reward model framework designed to align LLM role-play through continuous implicit preferences and rule-based validation.
• It leverages pairwise comparisons and transformer backbones to enhance narrative coherence, persona consistency, and multi-turn dialogue quality.
• Empirical results demonstrate significant performance gains over generic reward models, improving alignment in subjective and nuanced dialogue tasks.

RoleRM denotes specialized reward modeling frameworks designed to evaluate and align LLMs for profile-based role-play dialogue. It encapsulates two principal research programs in the literature: (1) the rule-based Role Reward Model in RAIDEN-R1, which applies verifiable and deterministic criteria to reward role awareness during RL fine-tuning, and (2) the continuous implicit preference-based RoleRM, which models human-like, fine-grained judgments of narrative and persona actuation via supervised preference ranking. Both approaches address the inadequacy of general reward models on highly subjective, multi-faceted role play tasks and introduce new metrics, datasets, and learning paradigms for rigorous alignment of LLMs in subjective dialogue scenarios.

1. Definition and Motivation

RoleRM, in the context of LLM-based conversational agents, refers to a reward model architecture and/or function specifically targeting evaluation and alignment in profile-based role play. Standard reward models, trained for QA or broadly defined “helpfulness,” fail to capture the layered, subjective dimensions required for high-fidelity persona enactment—such as narrative management, role consistency, stylistic fidelity, instruction following, and engagement. RoleRM solutions are motivated by repeatedly observed failures of generic RMs to provide meaningful feedback in role play: their predictions are often poorly correlated with expert human assessments, especially along narrative and stylistic axes (Ding et al., 11 Dec 2025).

2. Continuous Implicit Preferences (CIP) Framework

The RoleRM of (Ding et al., 11 Dec 2025) operationalizes subjective reward modeling by introducing the Continuous Implicit Preferences framework. CIP reframes reward supervision as a dense, pairwise comparison task using expert orderings, eschewing discrete or scalar scoring in favor of preference continuity. For each prompt-persona context, K candidate responses are generated (commonly K=5), then ranked in total order by annotators according to seven explicit role play competencies: narrative, multi-turn coherence, persona consistency, instruction following, scene transition, safety, and attractiveness.

From these rankings, several pair structuring approaches are applied:

• Neighbor Pairs (NEB): Compare only adjacent ranks $(y_i \succ y_{i+1})$.
• Best–Worst (BW): Compare top-to-bottom ranks.
• FULL: All possible pairs among K.

The core optimization maximizes Bradley–Terry likelihood over these pairs:

$$P(y_i \succ y_j | x) = \sigma(r_\theta(x, y_i) - r_\theta(x, y_j))$$

with the overall loss:

$$\mathcal{L}_{\mathrm{CIP}}(\theta) = -\mathbb{E}_{(x, y_i, y_j) \sim \mathcal{D}} [\log \sigma(r_\theta(x, y_i) - r_\theta(x, y_j))] + \lambda \|\theta\|_2^2$$

This ensures that reward values $r_\theta$ respect fine-grained role-play preferences.

3. RoleRM Architecture, Data, and Training

RoleRM is instantiated as a supervised reward model with a transformer backbone (Llama-3.1-8B-Instruct in (Ding et al., 11 Dec 2025)), and a single linear scoring head on the response’s last hidden state. Input representation concatenates a system prompt (specifying the character profile), full dialogue context, and candidate assistant completion. Training data comprises 35K multi-turn contexts from diverse role-play datasets (CoSER, RoleMRC, CharacterBench, CharacterEval) with five LLM-generated continuations per context, each ranked independently by three annotators under the CIP guidelines. Consensus- or majority-ranked pairs yield ~205K high-quality pairwise labels; an additional 150K pairs from open-domain datasets (re-annotated for role-play) augment diversity.

Preference structuring uses a hybrid “FULL-pure” strategy on consistency-filtered pairs. Training uses SGD with batch size 256, learning rate $9 \times 10^{-6}$, and weight decay $10^{-4}$ over two epochs.

4. Evaluation Methodology and Results (RoleRMBench)

Evaluation is standardized via RoleRMBench, a benchmark comprising seven granular role-play tasks (narrative, multi-turn coherence, consistency, instruction-following, scene transition, safety, attractiveness). For each (context, preferred $y^+$, rejected $y^-$) triple, correctness is assigned if $r_\theta(x, y^+) > r_\theta(x, y^-)$.

Model	Avg.	Nar	MT	Con	IF	Scn	Saf	Att
internlm2-20b-reward	70.6	70.4	68.3	67.6	76.0	72.7	66.1	75.0
GPT-4o-2024-08-06	69.1	66.7	66.7	66.9	71.0	68.2	78.8	67.6
CharacterRM	61.1	59.3	65.1	56.3	72.0	66.7	52.5	55.9
RoleRM (ours)	88.3	90.7	82.5	80.3	94.0	90.9	91.5	88.2

RoleRM achieves +24 percentage points over the best general-purpose RM (internlm2-20b-reward), with particularly pronounced improvements on narrative (+20.3 pp) and attractiveness (+13.2 pp). Over three seeds, this improvement is statistically significant ($p < 0.01$).

5. Rule-Based Verifiable Reward: RoleRM in RAIDEN-R1

A contrasting approach, RoleRM as implemented in RAIDEN-R1 (Wang et al., 15 May 2025), targets verifiability and RL tractability via a deterministic, rule-based reward function—Verifiable Role-Awareness Reward (VRAR). Instead of learning complex preference surfaces, VRAR returns a scalar in $\{0,1\}$ per sample based on exact or permissive role-specific key validation.

• Single-Term Validation (STV): Checks if a unique ground-truth keyword $k(x)$ appears in the generated response $y$.
• Multi-Term Dynamic Parsing (MTDP): For prompts where answers may vary semantically, a set $K(x)$ of acceptable forms is created; a Python function $f_x(y)$ returns True if $y$ matches any variant.

Formally:

$$R_\phi(x, y) = \max(R^\text{STV}_\phi(x, y), R^\text{MTDP}_\phi(x, y))$$

This reward is used in the Generalized Reward–Policy Optimization (GRPO) loop, a PPO-style method designed to maximize expected VRAR. Unlike preference-based RoleRM, VRAR offers absolute, reproducible feedback for RL fine-tuning.

6. Empirical Performance, Implications, and Limitations

RoleRM (CIP variant) demonstrates substantial gains in narrative coherence, persona and style fidelity, and holistic engagement. By leveraging multi-way comparisons and dense ranking signals, RoleRM learns a continuum of subjective preferences not captured by manually coded or generic reward signals (Ding et al., 11 Dec 2025).

The rule-based VRAR RoleRM approach, employed in RAIDEN-R1, translates discrete measures of role adherence into effective RL optimization signals, yielding superior Script-Based Knowledge (SBK) and Conversation Memory (CM) metrics relative to baseline and SFT-only models (e.g., 88.04% SBK, 88.65% CM for 14B-GRPO) (Wang et al., 15 May 2025).

Limitations for both approaches include:

• Current RoleRM (CIP) scale is limited to 8B backbone models; scaling to 70B+ expected to bring further improvement.
• Data volume for RoleRMBench is ~5K instances per subtask; expansion to longer, more varied, and multi-modal role-play anticipated.
• Pairwise loss does not directly model margins; listwise objectives may yield better preference surface calibration.

A plausible implication is that integrating CIP-based preference modeling with verifiable reward signals could yield even more robust role-play alignment by combining subjective perceptual signals with explicit factual correctness.

7. Significance and Future Directions

RoleRM, along with the CIP annotation paradigm and RoleRMBench, establishes a systematic methodology and open evaluation suite for subjective alignment of LLMs to nuanced, persona-centered communication tasks. This closes critical gaps observed in generic reward modeling, advancing model safety, coherence, and user-centered alignment in creative dialogue systems. Future work may investigate larger architectures, more intricate annotation schemes (e.g. margin-calibrated or listwise), and extensions into cross-modal and long-horizon storytelling modalities (Ding et al., 11 Dec 2025, Wang et al., 15 May 2025).