PERSONA Bench Benchmark

• PERSONA Bench Benchmark is a systematic framework that measures LLM sensitivity to persona-driven writing style variations.
• It employs automated persona-based rewriting and entailment filtering to produce over 1,200 stylistically diverse prompt variants.
• Using detailed performance metrics and bias analyses, it reveals significant disparities in LLM responses across different sociodemographic groups.

The PERSONA Bench Benchmark is a systematic framework for evaluating LLMs on their robustness to diversity in human writing styles, with the explicit goal of identifying and quantifying sensitivity to sociolinguistic and persona-driven prompt variation. By augmenting standard evaluation datasets with extensive persona-based prompt rewrites, it exposes instability and potential biases in LLM performance estimates, revealing shortfalls in current generalization beyond standardized or majority writing conventions. PERSONA Bench extends the typical scope of LLM benchmarking by linking model accuracy and reliability directly to linguistic, demographic, and sociocultural variation in prompts.

1. Benchmark Motivation and Definition

PERSONA Bench was introduced to address the external validity limitations of prevailing benchmarks, which predominantly use standardized, formal English and thus systematically under-represent the broad range of writing styles that naturally occur in real-world use. The core hypothesis motivating PERSONA Bench is that LLMs optimized on such monostylistic test sets may display brittle or uneven performance in the presence of sociodemographic and stylistic variation—posing risks to both real-world reliability and deployment equity. The benchmark formalizes the persona-based rewriting of test prompts, expanding each original context–question pair into many stylistic variants using systematically constructed personas (Truong et al., 29 Jul 2025).

2. Persona Construction and Writing-Style Diversification

PERSONA Bench begins by defining 100 “base personas” (sampled from PersonaHub), each described by occupation and intrinsic characteristics. Each base persona is then augmented with one attribute from each of four independent axes to create a total of 1,200 distinct personas:

• Native language: Chinese, English, Spanish
• Gender/sexual identity: male, female, LGBTQ+
• Education level: < high-school, high-school grad, college grad
• Age range: teenager, adult, elderly

Each persona collectively induces a distinct, automatically enacted writing style, affecting syntax, morphological complexity, lexical choices, and sentiment polarity. These styles are operationalized by prompting an LLM, in the role of each persona, to rewrite every evaluation prompt in its own linguistic register, under strict system constraints to avoid content drift (“Maintain all key information; do not add new content. Produce fluent English understandable to a general audience; if you cannot safely rephrase, reply ‘No.’”).

The pipeline automatically filters attempted rewrites that substantially alter the semantic entailment of original questions, ensuring ≥75% answer-consistency via entailment assessment. This procedure produces high coverage and writing diversity across task datasets.

3. Generation, Entailment Filtering, and Evaluation Pipeline

The rewriting process iterates over each original test example $(x_i, Q_i, A_i)$ and each persona $p$, producing rewritten prompt variants $x_{i,p}'$. Rewrites failing the entailment check (where fewer than 75% of associated $Q_i$ have answers preserved) are discarded. The resulting set $D'$ is the expanded, persona-augmented benchmark.

Evaluation of each LLM $m_j$ proceeds by running the model on each prompt variant $x_{i,p}'$ in $D'$, scoring accuracy (or a task-appropriate metric) against the gold-standard answers. The pipeline thus exposes the effect of persona-induced writing variation on LLM predictions, as well as enabling fine-grained analysis of per-persona performance bands.

Summary of the multi-step pipeline:

Step	Process Description	Filtering/Constraints
Persona enumeration	1,200 personas via 4-axes augmentation	Balanced across axes
Prompt rewriting	LLM in persona role rephrases each prompt	Forced to preserve information
Entailment check	LLM judge verifies retained answerability	≥75% questions retained per rewrite
Evaluation	Target LLMs scored on all accepted variants	Standard metrics (accuracy, etc.)

4. Metrics and Statistical Analyses

PERSONA Bench defines both aggregate and per-persona metrics, along with measures for fairness and ranking stability:

• Per-persona average: $$\hat p_p^j = (1/n_p)\sum_{i: (i,p)\in D'} f_{m_j}(x_{i,p}')$$
• Average across personas: $\bar p^j = (1/P)\sum_{p=1}^P \hat p_p^j$
• Variance across personas: $$\sigma_j^2 = (1/P)\sum_{p=1}^P (\hat p_p^j - \bar p^j)^2$$
• Weighted/post-stratified metrics: Adjusts for prompt-specific difficulty using k-means clustering and post-stratification; corrects for potential selection artifacts.

Model comparisons further employ non-parametric tests (Spearman $\rho$, Kendall’s $\tau$), sign tests for statistical significance, and analyses of linguistic correlates (Flesch reading ease, noun/verb ratios, hedge words). Benchmark leaderboards are recommended to report bands (min/max per persona) rather than single points, improving transparency regarding model robustness.

5. Empirical Findings: Sensitivity, Disparity, and Model Robustness

PERSONA Bench reveals that LLMs demonstrate substantial performance swings across persona-induced writing styles, even with the underlying semantic content held fixed. For widely used datasets:

• CoQA: LLM performance fluctuates by 27–55% (relative change) across personas.
• CosmosQA: 9–46% swings.
• DS-1000 (code generation): 43–80% swings.

Importantly, writing styles associated with “lower” sociodemographic groups (e.g., < high-school education, elderly) are over-represented in the lowest performance quartile (“global worst” personas). Lexical diversity is markedly increased versus standard evaluation: n-gram distinctness and cosine dissimilarity confirm that style variety is substantive, not superficial.

Linguistic correlates highlight that LLMs are more robust with complex, less “easy” prose—higher grade level, more complex clauses, fewer hedge words—suggesting potentially unintended optimization for formal style. Simulated leaderboard shifts can be dramatic (–19 to +14 ranks), underlining the risks of relying on monostylistic benchmarks for competitive or deployment decisions.

6. Implications, Deployment Risks, and Recommendations

Key implications of the PERSONA Bench results include:

• External validity concern: Standard benchmarks systematically over-estimate LLM capabilities in real-world, non-standard language scenarios.
• Equity risk: Users employing informal, elderly, or lower-education writing may face disproportionately degraded system performance.
• Methodological recommendations: Benchmarking should
  1. Integrate diverse, persona-augmented prompt variants alongside originals,
  2. Report performance as a range or weighted aggregate over personas (corrected for prompt difficulty),
  3. Supplement leaderboards with performance bands rather than point estimates,
  4. Pursue mitigation strategies such as style-robust fine-tuning and persona-aware calibration.

The PERSONA Bench pipeline is presented as a scalable, low-cost method for augmenting the external validity and inclusivity of future LLM evaluations, directly relevant for researchers, competitive benchmarking organizations, and practitioners assessing models for deployment across heterogeneous user populations.

7. Relationship to Broader Persona Benchmarking and Future Directions

While PERSONA Bench specifically addresses style-driven performance disparities, it complements benchmarks targeting other axes of personalization—such as preference alignment, long-term user modeling, memory-rich persona tracking, and pluralistic value alignment (cf. PersonaLens (Zhao et al., 11 Jun 2025), PersonaConvBench (Li et al., 20 May 2025), PersonaFeedback (Tao et al., 15 Jun 2025), PERSONA Bench for Pluralistic Alignment (Castricato et al., 2024)). Each tackles a distinct facet: PERSONA Bench uniquely decouples the effects of prompt style from reasoning or preference phenomena, isolating linguistic robustness as a primary concern.

Future work recommended in (Truong et al., 29 Jul 2025) includes:

• Further refinement of persona selection,

• Integration of human-in-the-loop validation,
• Cross-lingual persona benchmarking,
• Systematic comparison and mitigation of model vulnerabilities to style confounders,
• Longitudinal tracking of LLM evolution as benchmarks diversify.

PERSONA Bench thus constitutes an essential tool for surfacing latent brittleness in LLMs, with direct implications for continual evaluation, fairness auditing, and the pursuit of truly universal linguistic competence in foundational models.