Reliability Auditing for Downstream LLM tasks in Psychiatry: LLM-Generated Hospitalization Risk Scores

Abstract: LLMs are increasingly utilized in clinical reasoning and risk assessment. However, their interpretive reliability in critical and indeterminate domains such as psychiatry remains unclear. Prior work has identified algorithmic biases and prompt sensitivity in these systems, raising concerns about how contextual information may influence model outputs, but there remains no systematic way to assess these, especially in the psychiatric domain. We propose an approach for reliability auditing downstream LLM tasks by structuring evaluation around the impact of prompt design and the inclusion of medically insignificant inputs on predicted hospitalization risk scores, which is often the first downstream AI clinical-decision-making task. In our audit, a cohort of synthetic patient profiles (n = 50) is generated, each consisting of 15 clinically relevant features and up to 50 clinically insignificant features, across four prompt reframings (neutral, logical, human impact, clinical judgment). We audit four LLMs (Gemini 2.5 Flash, LLaMa 3.3 70b, Claude Sonnet 4.6, GPT-4o mini), and our results show that including medically insignificant variables resulted in a statistically significant increase in the absolute mean predicted hospitalization risk and output variability across all models and prompts, indicating reduced predictive stability as contextual noise increased. Clinically insignificant features had an effect on instability across many model-prompt conditions, and prompt variations independently affected the trajectory of instability in a model-dependent manner. These findings quantify how LLM-based psychiatric risk assessments are sensitive to non-clinical information, highlighting the need for systematic evaluations of attributional stability and uncertainty behavior like this before clinical deployments.

• The paper demonstrates how minor inclusion of irrelevant features significantly increases risk score variability in LLM-generated psychiatric assessments.
• It employs synthetic patient cohorts and varied prompt framings to quantify model instability using mixed-effects linear models.
• Results highlight the need for standardized prompt engineering and rigorous auditing to ensure fair, reliable clinical decision support.

Reliability Auditing for Downstream LLM Tasks in Psychiatry: Assessment of LLM-Generated Hospitalization Risk Scores

Introduction and Context

The application of LLMs in healthcare, particularly for psychiatric risk assessment, raises questions about interpretability, reliability, and bias—issues critically magnified in high-stakes, context-sensitive settings. While recent work has demonstrated strong diagnostic potential for LLMs in clinical psychiatry, there has been insufficient attention to how LLMs handle non-salient contextual information and prompt design variability when producing downstream risk assessments. This paper presents an empirical framework for systematically auditing the reliability of LLMs in generating psychiatric hospitalization risk scores, focusing on how irrelevant input features and variations in prompt framing affect model prediction stability.

Methodological Framework

A synthetic cohort of 50 patient profiles was constructed, each populated with 15 clinically relevant features and iteratively augmented with up to 50 additional clinically insignificant variables. This approach allowed for controlled probing of model sensitivity to non-clinical contextual noise. Four widely used LLMs (Gemini 2.5 Flash, LLaMa 3.3 70b, Claude Sonnet 4.6, GPT-4o mini) were evaluated. Each patient profile was presented under four distinct prompt framings: neutral, logical (logos), human impact (pathos), and clinical judgment (ethos).

For each (model, prompt, patient, feature configuration) combination:

• Risk scores were generated with stochastic repetitions ($n=5$) to measure intra-condition variability.
• Predictive instability was defined as the absolute deviation of the mean risk score from the clinically relevant baseline upon the addition of irrelevant features.

Linear mixed-effects models were used to quantify the effect of feature accumulation and prompt style on output instability. Prompt-feature interaction effects, prompt effects independent of feature count, and the trajectory of instability growth were rigorously tested with Bonferroni-corrected significance levels.

Main Findings and Numerical Results

Instability from Clinically Insignificant Features

Across 44,000 model-patient-prompt runs, the inclusion of non-clinical variables led to a statistically significant increase in mean predicted risk score deviation and output variance in nearly all model–prompt pairings. Notable empirical patterns include:

• GPT-4o mini: Significant instability across all prompt conditions with $p < 0.001$ for both simple and interaction effects.
• LLaMa 3.3 70b: Significant instability for neutral, logical, and clinical judgment prompts ($p<0.01$) with non-significant interaction effects.
• Claude Sonnet 4.6: Marked prompt-interaction effect under logical prompting ($p < 0.001$), suggesting model-specific susceptibility to prompt framing.
• Gemini 2.5 Flash: Instability was generally not significant except under the logical prompt ($p<0.001$ for prompt effect), indicating model-prompt idiosyncrasies.

Instability was largest following the initial batch of five irrelevant variables, plateauing with further additions. For all models, even minimal non-clinical noise could meaningfully perturb predicted hospitalization risk.

Prompt Sensitivity

Prompt framing exhibited independent and interaction effects on instability:

• The logical (logos) prompt consistently amplified instability, either as a level shift (Gemini, LLaMa) or through cumulative effects with increasing noise (Claude Sonnet, GPT-4o mini).
• Clinical judgment framing (ethos) was the most stable and least sensitive to cumulative contextual noise.
• No prompt style was universally optimal; prompt–instability interactions were model-dependent.

Attributive Auditing

Qualitative analysis revealed that models regularly assigned comparable and sometimes directionally significant attributions to non-clinical features as to core clinical indicators. Furthermore, models sometimes offered plausible-sounding rationales justifying the influence of behaviors or traits (e.g., donating blood, picking up litter), even when these features have no clinical correlation with psychiatric risk, raising concerns of value-laden or biased reasoning.

Practical and Theoretical Implications

Clinical Decision Support and Health Equity

The demonstrated sensitivity to non-clinical noise implies that LLM-based assessment tools may yield unreliable and inequitable outputs unless rigorous input and prompt constraints are enforced. Given real-world patient records often contain mixture of salient and non-salient information, even modest contextual noise can lead to non-trivial prediction shifts, potentially worsening disparities if irrelevant demographic or lifestyle data correlate with protected attributes.

Model Evaluation and Regulatory Implications

• Prompt engineering must be formalized as a controlled, methodological parameter in LLM pipeline design and audit.
• Pre-deployment auditing with adversarial noise and prompt perturbations should become a baseline requirement for AI healthcare systems.
• Reporting on the attributive behavior of output rationales should be mandated for transparency and bias detection.

Future Technical Directions

Key limitations—such as the black-box nature of LLM attribution, synthetic data constraints, and lack of missing data scenarios—highlight several future pathways:

• Mechanistically grounded attribution entropy metrics to quantify distribution of model-assigned weights.
• Evaluations on real, unstructured EHR notes and longitudinal records to characterize generalizability to actual clinical environments.
• Fairness audits across patient subgroups to detect non-uniform instability and potential equity harms.

Conclusion

The study robustly demonstrates that LLMs deployed for psychiatric risk assessment are highly sensitive to both the inclusion of clinically irrelevant input features and variations in prompt framing. Instability rapidly increases upon introduction of minor contextual noise and is differentially modulated by prompt strategy across model architectures. Models may encode and rationalize non-evidence-based attributions, raising risks for biased or unreliable clinical decision support. These findings reinforce the necessity for systematic reliability audits encompassing prompt and input noise sensitivity prior to any downstream clinical deployment of LLMs, and inform standards for transparency and accountability in emerging AI-driven healthcare applications.

---

Reference:

Reliability Auditing for Downstream LLM tasks in Psychiatry: LLM-Generated Hospitalization Risk Scores (2604.22063)