PsychBench: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations

Abstract: LLMs are increasingly deployed to simulate patients for clinical training, research, and mental health tools, yet population-level validity remains largely untested. We introduce PsychBench, the first epidemiological audit of LLM patient simulation: 28,800 profiles from four frontier models (GPT-4o-mini, DeepSeek-V3, Gemini-3-Flash, GLM-4.7) evaluated against NHANES and NESARC-III baselines across 120 intersectional cohorts. The central finding is a coherence-fidelity dissociation: models produce clinically plausible individuals while misrepresenting the populations they are drawn from. Variance compression ranges from 14 percent (GLM-4.7) to 62 percent (DeepSeek-V3), eliminating the distributional tails of clinical reality. Despite test-retest correlations above r = 0.90, 36.66 percent of cases cross diagnostic thresholds between runs. Symptom correlation matrices diverge across demographic groups beyond split-half noise, with transgender populations diverging three to five times more than racial differences. Calibration bias is systematic and asymmetric. Models overestimate depression severity for most groups by 3.6 to 6.1 points (Cohen d = 1.13 to 1.91), consistent with training on clinical corpora with elevated base rates. For transgender women the direction inverts: models capture only 8 to 46 percent of documented minority stress elevation, yielding a -5.42 residual (d = -1.55). Models also attribute irritability to Black men and fatigue to women beyond matched controls, encoding racialized and gendered assumptions. Patterns replicate across US and Chinese architectures, indicating failures tied to current training paradigms rather than isolated implementations. For most users, LLM mental health tools risk pathologizing ordinary distress; for transgender users, algorithmic erasure of genuine need. The patients look right. They do not represent real populations.

• The paper demonstrates that LLM simulations achieve high internal coherence yet fail to replicate real-world epidemiological variance across demographic intersections.
• The study employs standardized clinical instruments and network divergence metrics to reveal significant threshold instability and structural biases in simulated profiles.
• The audit highlights calibration bias and variance compression, underlining the risks of deploying LLMs for clinical mental health assessments without demographic-aware validation.

PsychBench: A Systematic Audit of Epidemiological Fidelity in LLM Mental Health Simulations

Introduction and Scope

"PsychBench: Auditing Epidemiological Fidelity in LLM Mental Health Simulations" (2604.17359) establishes the first large-scale, systematic epidemiological audit of LLM-based simulated patient populations within psychiatric assessment contexts. The research addresses a central, underexplored question: while LLMs can generate clinically plausible synthetic patients—profiles that adhere to diagnostic logic and surface-level realism—do cohorts of such simulated patients faithfully reproduce the statistical, network, and intersectional properties of true population epidemiology? The answer, the study demonstrates, is negative, with implications that challenge the current validation and deployment paradigms for LLM-based clinical tooling.

Methodological Overview

The audit leverages four contemporary LLMs—OpenAI's GPT-4o-mini, DeepSeek-V3, Gemini-3-Flash, and GLM-4.7—subjected to 28,800 simulated psychiatric profiles derived from 120 intersectional demographic cohorts. Profiles are scored with standardized instruments (PHQ-8, GAD-7, AUDIT-C, PCL-5), and evaluated against NHANES and NESARC-III baseline distributions. Two prompt framings are used: a clinical, third-person input representative of professional deployment and a personal, first-person narrative relevant to direct user interaction, capturing context sensitivity.

Coherence-Fidelity Dissociation

A major contribution is the formalization and empirical quantification of the coherence-fidelity dissociation. All examined models exhibit near-perfect internal coherence: individual simulated patients are clinically plausible, pass instrumented DSM-5 logic checks (zero gateway violations in 28,714 profiles), and maintain high test-retest correlation (r > 0.90). However, at the population level, statistical properties are systematically misrepresented. The study introduces the Stereotype Index (SI), measuring the ratio of model-generated to ground-truth standard deviation within demographic cohorts. Massive variance compression is observed, with SI as low as 0.38 (DeepSeek-V3), reflecting truncation of distributional tails critical for clinical realism.

Figure 1: Stereotype Index heatmap across demographic intersections; severe variance compression is evident, particularly along the socioeconomic axis.

Standard validation approaches—coherence checks, individual plausibility, and correlation measures—fail completely to detect these population-level failures. This finding is visually summarized in the dissociation between individual-case coherence and overall fidelity:

Figure 2: All models achieve uniformly high coherence at the individual level while exhibiting large, variable deficits in population-level fidelity as assessed by mean Stereotype Index across core categories.

Threshold Instability and Stochasticity

Despite high continuous-level reliability, diagnostic stability at categorical cut points is substantially compromised. The audit finds a 36.66% threshold flip rate between matched runs, notably at clinically significant boundaries (e.g., the PHQ-8 mild/moderate depression threshold).

Figure 3: Heatmap reveals high frequency of diagnostic transitions between runs, even with high test-retest correlation, with instability notably concentrated at critical category boundaries.

This instability is not a consequence of increased stochastic variance for minority populations; demographic groups exhibit statistically indistinguishable run-to-run variance.

Figure 4: Demographic intersections show uniform run-to-run stochasticity, confirming that categorical instability is deterministic rather than noise-driven.

Network Structural Divergence

The study employs item-level correlation matrix analysis to uncover latent structure divergence—a direct test of whether simulated mental health constructs reflect universal or group-specific symptom networks. Frobenius norms reveal structural distances far exceeding noise thresholds, with divergence for transgender populations (d_F = 1.14) outstripping all racial differences 3–5×.

Figure 5: Symptom network structures for minority groups (particularly transgender) diverge beyond the established noise ceiling, documenting nontrivial representational shifts across demographics.

Permutation-based controls reinforce that these divergences are genuine and not artifacts of sampling variability.

Figure 6: Observed structural divergences remain well beyond the empirical noise floor determined via permutation, indicating true model-internal network differentiation.

Calibration Bias and Stereotyping

Models systematically overestimate depressive symptom severity for most groups (+3.6 to +6.1 PHQ-8 points; Cohen’s d = 1.13–1.91), likely due to clinical-corpus skew. In stark contrast, transgender women are underestimated by –5.4 points (d = –1.55), capturing as little as 8–46% of the epidemiologically documented minority stress effect. Furthermore, symptom-level attribution is stereotyped: Black men are assigned irritability and women fatigue at exaggerated rates, even after adjusting for aggregate severity.

Figure 7: Bias residuals show across-the-board overestimation for most groups and categorical underestimation for transgender women, highlighting both uniform and asymmetric calibration errors relative to population data.

Model Comparison and Regional Influences

Models differ quantitatively in both variance preservation and stability. GLM-4.7 best preserves variance but shares instability; DeepSeek-V3 is most stable but maximally compresses variance. No model balances fidelity and stability. Calibration of culture-specific epidemiological features (e.g., the Asian Paradox) is achieved only by Chinese-developed models, indicating direct influence of corpus and alignment protocols; US-developed models miscalibrate in divergent directions.

Figure 8: Inter-model correlation structure reveals tighter behavioral clustering among China-based models compared to US-based models.

Practical, Theoretical, and Regulatory Implications

The audit rigorously invalidates the assumption of surrogate population fidelity for LLM-generated clinical cohorts. RLHF and safety alignment create population-level distortions—variance compression, network divergence, asymmetric suppression of minority distress—that are invisible at the individual-case or surface-metric level. The findings portend widespread deployment risks: for most users, inflated pathologization of ordinary distress; for transgender users, algorithmic erasure of true clinical risk; for clinicians, exposure to narrow, non-epidemiological case distributions with consequences for training and resource allocation.

Theoretically, the work highlights alignment’s unintended byproducts: algorithms optimized for harmlessness can erase valid signals from high-risk populations, introducing systematic harm under the guise of safety. These mechanisms empirically extend the "harm erasure" hypothesis and advocate for fidelity metrics (e.g., SI, network divergence) as required validation complements to standard coherence checks.

Practically, recommendations include targeted variance augmentation, demographic- and context-stratified validation, independent sampling parameter optimization, and subgroup harm audits as necessary for responsible clinical model governance. Regulatory paradigms, currently focused on aggregate accuracy and refusal/hallucination rates, are shown inadequate to guarantee representation and non-discrimination for protected demographic strata.

Future Prospects in AI Auditing and Clinical Simulation

Future research must (1) develop training and alignment protocols that preserve demographic- and intersectional-level variance and correlation structure, (2) extend calibration auditing to other clinical domains, (3) integrate temperature/sampling-based regularization as a distributional safeguard, and (4) establish standardized subgroup auditing and transparent reporting as prerequisites for clinical approval. Collaboration between clinical, epidemiological, and AI research sectors is essential to ensure synthetic populations reflect real-world complexity and heterogeneity.

Conclusion

This audit demonstrates that current frontier LLMs produce psychiatric patient simulations with high internal coherence but fundamentally compromised epidemiological fidelity. These models compress variance, induce network and threshold instabilities, encode both overpathologization and algorithmic erasure, and entrench demographic stereotypes—failures not detectable via standard validation or surface plausibility review. The technical fixes are visible: variance-aware alignment, demographic and context-aware validation, and structural fidelity metrics. Until implemented, use of such models for clinical simulation or mental health assessment should be regarded as systematically biased and unrepresentative, with substantial ethical, scientific, and societal implications.