Impact of user communication diversity on agent performance and success

Quantify how variation in user communication styles, linguistic backgrounds, and cultural norms affects large language model agent performance and task success in task-oriented conversational settings, including dimensions such as formality, verbosity, and politeness norms.

Background

Real users differ along multiple communicative dimensions and sociolinguistic backgrounds, which can influence how they interact with agents and the information the agent receives across turns.

The paper highlights that current simulations often overlook this diversity and later shows disparate success rates and calibration errors across dialects and age groups, underscoring the need to precisely quantify how such diversity impacts agent outcomes.

References

For example, even in a simple retail assistance scenario, users might vary along dimensions such as formality, verbosity, and politeness norms—but it remains unclear how much this diversity meaningfully impacts agent performance and task success \citep{truong-etal-2025-persona}.

— Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations  (2601.17087 - Seshadri et al., 23 Jan 2026) in Introduction