ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?

Abstract: Role-playing language agents (RPLAs) should play characters whose values and behavior evolve as the story progresses, not maintain a fixed persona. Existing benchmarks measure factual recall at a given chapter, not whether responses align with the character's psychological trajectory, especially in scenarios the source text never explores. We introduce ArcANE (Arc-Aware Narrative Evaluation), an automatically constructed benchmark spanning 17 novels and 80 principal characters. A Character Arc segments the narrative into phases along a psychological axis, and each probe poses the same scenario across phases, spanning both situations within the source text and situations beyond it. Across six models and six context modes, conditioning on the Character Arc tops every other context strategy on every model, and the gap is largest on scenarios outside the source text where retrieval has nothing to find. We further fine-tune open-weight models on the same data to obtain ArcANE-8B/32B, which widen the Arc advantage even more on scenarios outside the source text.

• The paper introduces ArcANE, a benchmark that evaluates RPLA adherence to evolving character arcs through automated arc extraction and multi-phase probing.
• It details a pipeline leveraging character events and psychological axes, validated by human annotators, to ensure robust and context-sensitive performance.
• Experimental results demonstrate that arc-grounded contexts significantly improve model fidelity, particularly in Out-of-World scenarios.

ArcANE: Benchmarking Temporal Behavioral Fidelity in Role-Playing Language Agents

Motivation and Problem Statement

Role-playing language agents (RPLAs) are increasingly prevalent in entertainment, education, and simulation, demanding not only fluent dialogue generation but sustained, immersive character fidelity. Existing benchmarks for RPLA evaluation typically focus on surface traits, static persona reproduction, or factual recall at fixed points in the narrative. However, robust character simulation in narrative contexts requires the agent to express evolving psychological states as the story unfolds, particularly in scenarios not explored by the source text. ArcANE addresses this gap by introducing a new benchmark for temporal behavioral fidelity: does an RPLA stay in character at the right moment, expressing the character’s psychological trajectory in both canonical and novel situations?

Benchmark Construction: Character Arcs and Probe Generation

ArcANE operationalizes narrative-aware evaluation through automated extraction of Character Arcs and multi-phase probes from novels. The construction pipeline, shown in (Figure 1), consists of event and state streams per character, which independently induce intrapersonal (beliefs, motives) and relational (trust, esteem) axes grounded in psychological and literary theory. These axes are reconciled through LLM ensembles and validated by human annotators to ensure external validity and coverage.

Figure 1: The ArcANE pipeline: constructing character arcs (top) and evaluation probes (bottom).

A Character Arc aligns the character’s key events with phase-segmented descriptions along an explicit psychological axis (e.g., Harry Potter’s arc from “Punitive Justice” to “Empathic Forgiveness”). Each arc yields probes consisting of one scenario-question pair posed across all phases. Probes are categorized as In-Scenario (verbatim source scene), In-World (fictional but plausible within the source’s setting), or Out-of-World (transposed to an era outside the source text). The latter force models to generate behavior based solely on arc-derived psychological state.

Figure 2: An example from ArcANE: an Out-of-World probe elicits different moral responses from Harry across his Character Arc, shifting from Punitive Justice in Book 1 to Empathic Forgiveness in Book 5.

Experimental Protocol and Evaluation Metrics

ArcANE covers 17 novels, 80 principal characters, 544 arcs, and 4601 probes. Evaluation is conducted across six models (open-weight and proprietary) and six context strategies, including Vanilla (no context), Summary, RAG (retrieval-augmented), LifeChoice, TimeChara, and Arc (providing only the truncated character arc at query chapter). The primary focus is the context-strategy axis: which context most robustly grounds the model in the correct phase.

Response evaluation uses an LLM judge, scoring on:

• APF (Action Phase-Fidelity): overt action alignment,
• RPF (Reasoning Phase-Fidelity): mechanism-based reasoning,
• RAE (Reasoning-Action Entailment): coherence between reasoning and action,
• PTF (Phase Trajectory Fidelity): aggregate trajectory alignment across phases.

Strong Claims and Numerical Results

Across all models, arc-grounded context consistently produces the strongest overall scores. The Arc context advantage amplifies with scenario source distance: minimal lift in In-Scenario, substantial lift in In-World, and maximal lift in Out-of-World, where retrieval adds no value and only phase-anchored arc information enables plausible generation. For example, DeepSeek-V4-Pro shows a +7.7 point overall gap for Out-of-World probes relative to the best non-Arc baseline.

Figure 3: Per-category Arc lift (Arc minus best non-Arc) for added RPLA baselines and the ArcANE models. SFT/DPO at each size are adjacent.

Additionally, models fine-tuned on ArcANE (ArcANE-8B, ArcANE-32B) outperform base models on all probe types, especially on Out-of-World probes. DPO training increases temporal sensitivity, further widening the trajectory-alignment gap. Ablation studies show arc-phase content is essential: mixed-arc (wrong character's arc) actively misleads trained models and compressed arc contexts (ArcHint) only partially recover the Arc lift, demonstrating the necessity of per-phase prose for learned arc tracking.

Figure 4: Arc source-of-effect ablation across three models, validated subset. DS refers to DeepSeek.

Validation, Robustness, and Analysis

Human annotation and cross-LLM judge replication establish the robustness of the evaluation protocol. Majority plausibility for judge scores is 87.1%, and re-scoring correlations with human annotators are $r = 0.96$ on overall metrics. Perturbation experiments demonstrate the sensitivity of the PTF metric to phase-order corruption, especially in models trained on ArcANE, confirming behavioral phase-tracking evaluation rather than superficial resummarization.

Figure 5: Human Annotation Page for axis rating.

Figure 6: Induced axes are clustered and grounded against established literary or psychological scholarship, following the construct-grounding criterion of Values in the Wild~\cite{huang2025values}.

Character-class breakdown (central vs. supporting) indicates Arc grounding benefits low-prevalence characters more than high-prevalence ones, counter to memorization-driven explanations.

Figure 7: Arc-over-Vanilla Overall lift, central ($N{=}$) vs supporting characters.

Practical and Theoretical Implications

ArcANE demonstrates that current RPLA benchmarks neglect temporal behavioral fidelity, and that conditioning on explicit phase trajectories is critical for generating context-sensitive behavior across narrative time. Retrieval-based and persona-only methods fail under out-of-distribution scenarios or low-popularity titles, while arc-grounded context enables robust generalization. This has broad implications for simulation, education, and entertainment applications where character evolution and situational contingency are critical. The pipeline also introduces a compact axis-index hint for inference on general-purpose models in long-context or multi-character deployments.

Future Directions

The next frontier for ArcANE and RPLA evaluation is interactive, multi-turn dialogue where arc progression emerges from agent-user interaction. Incorporating user-character and character-character interaction dynamics will be necessary for simulating richer social worlds and validating agent fidelity in more complex, adaptive contexts. Extending arc-based evaluation beyond novels, to scripts, games, and other narrative media, is promising for generalizing narrative AI techniques.

Conclusion

ArcANE establishes a new standard for evaluating whether role-playing language agents track and express evolving character state at the right narrative moment. Conditioning on character arcs at inference time yields substantial lift, especially in open-ended, non-source scenarios. The benchmark and post-trained models demonstrate clear temporal sensitivity and trajectory fidelity, offering a scalable platform for future developments in narrative-aware role-playing AI.