MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignment

Abstract: User simulators are essential for the scalable training and evaluation of interactive AI systems. However, existing approaches often rely on shallow user profiling, struggle to maintain persona consistency over long interactions, and are largely limited to English or single-domain settings. We present MUSE, a multi-domain Chinese user simulation framework designed to generate human-like, controllable, and behaviorally consistent responses. First, we propose Iterative Profile Self-Evolution (IPSE), which gradually optimizes user profiles by comparing and reasoning discrepancies between simulated trajectories and real dialogue behaviors. We then apply Role-Reversal Supervised Fine-Tuning to improve local response realism and human-like expression. To enable fine-grained behavioral alignment, we further train a specialized rubric-based reward model and incorporate it into rubric-guided multi-turn reinforcement learning, which optimizes the simulator at the dialogue level and enhances long-horizon behavioral consistency. Experiments show that MUSE consistently outperforms strong baselines in both utterance-level and session-level evaluations, generating responses that are more realistic, coherent, and persona-consistent over extended interactions.

• The paper introduces MUSE, which uses iterative profile self-evolution and role-reversal SFT to simulate authentic multi-turn Chinese user behaviors.
• It leverages a rubric-guided reward model with reinforcement learning to achieve high persona consistency, dialogue coherence, and natural language style.
• Experimental evaluations across six domains show MUSE significantly outperforms baselines in metrics like AI-generated probability and style similarity.

MUSE: Multi-Domain Chinese User Simulation via Self-Evolving Profiles and Rubric-Guided Alignment

Motivation and Background

The scalable development and evaluation of interactive AI systems mandate robust user simulation capabilities, especially as human-in-the-loop approaches become increasingly untenable. Prior user simulators have exhibited critical limitations: reliance on hand-crafted rules or single-pass LLM extraction produces oversimplified user profiles lacking nuanced behavioral fidelity; long-term persona consistency and contextual coherence degrade across multi-turn exchanges; and coverage remains predominantly monolingual (English-centric). MUSE addresses these deficits by proposing a framework for multi-domain Chinese user simulation that synthesizes nuanced persona profiles and applies rubric-guided alignment to optimize behavioral consistency over extended interactions.

Methodological Framework

Iterative Profile Self-Evolution (IPSE)

MUSE introduces IPSE for user profile refinement, which is grounded in iterative dialogue reconstruction and discrepancy-aware reasoning. Starting with an initial profile extracted from real Chinese multi-turn dialogues, IPSE employs profile-conditioned LLM rollouts to simulate interaction trajectories. These are then critiqued using a reasoning model that diagnoses behavioral deviations and profile deficiencies, progressively updating the profile to more accurately induce authentic user behavior consistent with the observed ground truth. This approach demonstrably prevents information leakage and ensures temporal logic preservation within user strategy, surpassing the capabilities of single-pass profile extraction methods.

Figure 1: MUSE training pipeline overview, detailing the flow from IPSE profile extraction to rubric-guided multi-turn RL.

Role-Reversal Supervised Fine-Tuning (SFT)

To achieve local realism in simulated user responses, MUSE employs Role-Reversal SFT: rather than mapping user prompts to assistant responses, the model is trained to predict user utterances conditioned on the optimized profile and dialogue history. This configuration enhances expression alignment and crafts responses that exhibit the linguistic and expressive patterns of genuine human users.

Rubric-Guided Reward Model and Multi-Turn Reinforcement Learning

For fine-grained behavioral supervision, MUSE incorporates a reward model trained on expert-annotated rubric dimensions: human-likeness, persona consistency, and context coherence. The reward model leverages a two-stage process: supervised warm-up captures rubric criteria via Chain-of-Thought (CoT) reasoning and scoring standards, followed by RLVR-based calibration utilizing distance-aware rewards for scalable, granular feedback. This reward signal is used within a rubric-guided multi-turn RL stage (employing GRPO) to optimize session-level consistency and robustness in user simulation, focusing particularly on challenging dialogue samples to maximize behavioral fidelity.

Experimental Validation

Dataset Composition

MUSE's evaluation spans six Chinese dialogue domains: General Chat, Customer Service, Medical, Legal, Sports & Entertainment, and Technology & Education. Sessions are selected for richness and minimum turn depth to enable substantive persona manifestation and long-term interaction analysis.

Evaluation Metrics

Performance is measured at both utterance and session levels. Utterance metrics include: AI-generated probability (lower is more human-like), style similarity, author verification accuracy, and LLM-judge criteria (contextual relevance, response fidelity, goal contribution, linguistic naturalness). Session-level metrics examine persona consistency, goal-oriented effectiveness, dialogue coherence, and constraint compliance, providing a holistic measure of simulated behavior across sustained conversations.

Numerical Results

MUSE achieves strong numerical superiority on both granularities. Utterance-level AI-generated probability drops to 31.18 (versus 45.50 for GPT-4o); style similarity and AVA reach 0.7534 and 64.89, respectively. LLM-judge metrics (contextual relevance, response fidelity, goal contribution, linguistic naturalness) are also maximized, with session-level persona consistency at 0.8378 and dialogue coherence at 0.8685, outperforming GPT-4o and other baselines.

Ablation studies confirm the effectiveness of IPSE (reducing profile perplexity), the critical role of the progressive SFT-RLVR pipeline in reward model accuracy (0.9067 overall accuracy with RL/CoT), and the indispensable role of multi-turn RL for long-horizon behavioral coherence. Contradictory to conventional wisdom, the results substantiate that strong utterance-level generation alone is insufficient for robust persona maintenance, necessitating RL-driven alignment.

Implications and Future Directions

Theoretically, MUSE advances the paradigm of user simulation by jointly modeling profile quality, local response realism, and trajectory-level behavioral alignment, moving beyond simplistic persona extraction and superficial role adherence. Practically, MUSE provides a framework for agent training, synthetic data generation, and automated evaluation that is scalable, controllable, and behaviorally authentic across multiple Chinese domains. The model’s rubric-guided approach sets a precedent for leveraging fine-grained human supervision in behavioral model alignment, which can be adapted for non-Chinese languages and more specialized interaction scenarios.

Future directions include cross-lingual transfer, dynamic rubric adaptation for unseen domains, and unconstrained long-horizon simulation with flexible context scaling, all of which will be necessary for generalizing user simulators to broader real-world applications and mitigating biases from expert annotations.

Conclusion

MUSE establishes a robust methodology for multi-domain Chinese user simulation, integrating self-evolving profile refinement, role-reversal expression alignment, and rubric-guided RL optimization. Empirical results demonstrate that MUSE generates user behaviors with strong persona fidelity, coherence, and controllability over extended interactions, eclipsing state-of-the-art baselines. Its modular approach facilitates reproducible synthetic user generation suitable for agent training, evaluation, and synthetic dialogue data creation, with substantial implications for multi-domain conversational AI and task-oriented simulation (2604.13828).