CharacterDial: Character-Based Dialogue Systems

• CharacterDial is a framework for generating dialogue grounded in rich character profiles, ensuring long-range persona consistency.
• It employs methods like masked dialogue generation and persona fusion with Transformer-based models to encode identity, behavior, and emotion.
• The approach underpins applications in narrative generation, role-playing games, and virtual agents, validated through specialized datasets and targeted metrics.

Character-based dialogue (CharacterDial) refers to the generation, understanding, and evaluation of dialogue that is explicitly grounded in diverse, richly-parameterized character profiles—identities, behaviors, emotional attributes, and social contexts—rather than merely producing generic, context-independent responses. This paradigm underlies role-playing conversational AI, personalized agents, and narrative systems that require models to maintain long-range persona, behavioral, or emotional consistency, especially in scenarios such as story-telling, gaming, or open-ended social interaction.

1. Foundations and Problem Formulation

CharacterDial research targets two core challenges: ensuring generated dialogue consistently reflects a character’s defining features, and enabling accurate speaker attribution or persona recognition in multi-character settings. Early formalizations, such as in "A Benchmark for Understanding and Generating Dialogue between Characters in Stories," instantiated this via:

• Masked Dialogue Generation (DialGen): Given a story $S = (s_1, \dots, s_T)$ with $K$ masked turns, generate the missing dialogue $D = (d_1, \dots, d_N)$ to maximize $P_\theta(D|S_\text{masked})$, factoring character coherence and informativeness into the generative objective.
• Dialogue Speaker Recognition (DialSpk): Given $S$, a character set $C = \{C_i\}$, and $M$ masked dialogue spans $D_1, \dots, D_M$, predict $i_m$ such that $D_m$ is attributed correctly to $C_{i_m}$, using $P(\hat{Y}_m = C_i|S)$ (Yao et al., 2022).

Methodologically, these are framed as conditional language modeling and multi-class classification tasks, but with explicit conditioning on or prediction over structured character representations rather than global context alone.

2. Character Representation and Model Architectures

State-of-the-art systems employ Transformer-based architectures (e.g., autoregressive LLMs, BERT/BART) augmented with modules that encode character-centric features to control and track persona state throughout dialogue. Approaches include:

• Explicit Character Embedding: Aggregation of encoder layer hidden states $h_k$ where token $s_k$ mentions $C_i$ (mean-pooled via a character encoder such as a 1-layer Bi-Transformer), projected to a character vector $C_i$.
• Persona Fusion: At decoding step $n$, select the persona vector most proximal (e.g., via cosine similarity) to the current hidden state $\hat{H}_n$ and inject it into generation, typically through late fusion and nonlinear transformation (e.g., SiLU + LayerNorm followed by softmax prediction).
• Profile-Conditioned Prompting: Serialize and prepend multi-field character profiles (identity, interests, viewpoints, behavior, emotion, etc.) to each conversational context, as in CharacterGLM (Zhou et al., 2023), with no architectural changes required beyond input formatting.
• Multimodal Fusion: In systems requiring visually grounded dialogue (e.g., Action2Dialogue (Kang et al., 22 May 2025)), visual-semantic features from frames (using encoders such as BLIP) are linearly combined with prompt embeddings and dialogue memory to produce dialogue reflecting both persona and scene (Kang et al., 22 May 2025).

This enables not only persona retention but situational adaptation—characters react consistently to both evolving context and physical environment.

3. Datasets and Benchmarking Resources

Comprehensive, large-scale datasets with annotated character profiles form the empirical backbone for evaluation and model development:

Dataset	Size	Language(s)	Character Coverage	Distinct Features
DialStory	105k stories	Chinese	Avg. 3.5 chars/story (NER-extracted)	Autolabeled dialogue, speaker, plot coherence (Yao et al., 2022)
RoleplayPref	16,888 dialogues	Chinese/English	1,108 chars (13 subcats)	Multi-turn, LLM-generated, preference pairs (Fang et al., 29 May 2025)
CharacterBench	22,859 samples	Chinese/English	3,956 chars (25 subcats)	11 fine-grained eval dimensions, dense/sparse labels (Zhou et al., 2024)
MCPDial	269 convs	English	250 NPC + 750 Player personas	Game-grounded, persona-rich, function calls (Alavi et al., 2024)
Released CharacterGLM subset	1,034 dialogues	Chinese	250 characters	Human role-playing, annotated profiles (Zhou et al., 2023)

These datasets provide structured context—character profile, dialogue history, and, where relevant, multimodal cues—enabling targeted probing of character centricity, memory, factuality, and emotional fidelity. Some, such as CharacterBench, further segment evaluation into sparse (feature-extracted) and dense (per-response) dimensions for diagnostic granularity (Zhou et al., 2024).

4. Evaluation Protocols and Metrics

Evaluation of CharacterDial systems leverages both traditional generative metrics and specialized dimensions tailored to persona fidelity:

• Text Similarity/Diversity: BLEU-1/2, Distinct-n (intra-response diversity), BERTScore (semantic proximity).
• Coherence: Automated classifiers to distinguish coherent vs. shuffled story+dialogue samples (e.g., fine-tuned BERT).
• Speaker Attribution Accuracy: Dialogue-level and story-level speaker accuracy, as in DialSpk; human upper-bound ≈98% (Yao et al., 2022).
• Persona Consistency and Customization: Human-judged or automated scoring along axes such as memory consistency, attribute/behavior alignment, boundary/factual accuracy, empathy, engagement, and morality (cf. CharacterBench’s 11 dimensions and scales) (Zhou et al., 2024).
• Automatic Judge Correlation: Specialized automatic judges (e.g., CharacterJudge, Qwen2-7B-Chat-based) showing Pearson correlation 68% (zh), 64% (en) with human annotations, outperforming GPT-4-based judges (Zhou et al., 2024).
• Longitudinal and Multimodal Evaluation: Part-turn, full-dialogue, and multimodal (video/speech) metrics—e.g., CLIPScore for visual–verbal alignment, dynamic time warping for speech–audio congruence (Kang et al., 22 May 2025).
• Preference Modeling: Paired ranking accuracy with act-adaptive margin models (ChARM) yielding 13% improvement over Bradley–Terry baselines on RoleplayEval (Fang et al., 29 May 2025).

Manual annotation, pairwise ranking, and scenario-specific spot checks (as in user studies or targeted queries) remain crucial for high validity, especially given the sparsity and variability of persona expression across turns.

5. Advances in Learning: Optimization and Reward Modeling

Recent systems address limitations in generalization, annotation cost, and persona adaptability using advanced learning algorithms:

• Act-Adaptive Margin Reward Learning: For character-based conversational preference modeling, ChARM introduces a margin based on KL divergence between the model output and preferred/non-preferred responses, dynamically scaling the separation based on model confidence ($\mathcal{M}(\theta)$). This yields better discrimination over "acts" (contexts, roles) without over-penalizing ambiguous samples (Fang et al., 29 May 2025).
• Self-Evolution Mechanism: Unlabeled data is mined and filtered using the existing reward model and act-adaptive margin, with hard/difficult pairs rewritten using LLMs, iteratively enriching training data at low annotation cost. Accuracy increased from 58.8% to 62.1% in 4 loops on a 2,250-sample seed (Fang et al., 29 May 2025).
• Direct Preference Optimization (DPO): Used in conjunction with judged samples to fine-tune conversational LLMs, as implemented in CharacterBench and ChARM pipelines, leading to empirically demonstrated performance improvements in pairwise competitions and aspect-level scores (Zhou et al., 2024, Fang et al., 29 May 2025).

These methods enable stable, scalable adaptation of reward models and dialogue agents to evolving user preferences, profile variations, and context diversity.

6. CharacterBench, Evaluation Dimensions, and Key Insights

CharacterBench provides a granular evaluation suite, distinguishing eleven key dimensions by sparsity (feature-dependent/dense) and aspect (e.g., Memory, Knowledge, Persona, Emotion, Morality, Believability). Tailored queries are constructed to force targeted feature manifestation for sparse dimensions, addressing the inefficiency of waiting for persona aspects to arise spontaneously in open-ended dialogue (Zhou et al., 2024). Principal findings include:

• General-purpose LLMs outperform specialized role-play LLMs; large open-source models match closed-source performance in overall customization.
• Aspect weaknesses: Fact accuracy, emotional nuance, and engagement remain consistently lower than safety or profile memory.
• Automatic evaluation with CharacterJudge achieves rank correlation with human judgments (Spearman ρ = 73.1%) well above prior benchmarks.

This suggests robust, fine-grained generative evaluation is necessary for guiding research on authentic, engaging character-based dialogue.

7. Applications and Future Directions

CharacterDial systems span interactive storytelling, role-playing agents, gaming, and visually/physically grounded human–AI collaboration. Notable applications include:

• Narrative Generation: Story plot advancement with multi-character interaction and evolving persona trajectories (DialStory, Action2Dialogue).
• Game Dialogue: Persona-grounded NPC and player conversations with canonical function call integration (MCPDial).
• Social Role-Playing and Virtual Companions: Realistic, emotionally nuanced exchanges with user-customized characters (CharacterGLM, RoleplayPref).

Promising future research directions outlined include dynamic character state tracking, personality and emotion modeling beyond static profiles, multimodal integration for grounded dialogue, and reward modeling for richer, multi-dimensional alignment (e.g., with respect to plot coherence and emotional depth) (Yao et al., 2022, Fang et al., 29 May 2025, Zhou et al., 2023).

Ongoing challenges noted are the difficulty of sustaining long-range persona consistency, achieving high factual reliability, surfacing emotional content, and aligning automatic judge models robustly with human-level requirements. As benchmarks, datasets, and learning paradigms continue to evolve, CharacterDial remains a central and rapidly advancing research area at the intersection of natural language generation, control, and human-centered AI.