- The paper introduces a clinician-constructed, weighted rubric to dissect LLMs' atomic clinical inferences across complex scenarios.
- It reveals that while models excel in superficial tasks, they consistently fail on high-priority criteria, with essential pass rates as low as 32%-42%.
- The study demonstrates that LLM-based autoraters achieve over 92% agreement with expert judgments, yet significant gaps in clinical reasoning and safety remain.
Rubric-Based Controlled Evaluation of LLMs in Clinical Reasoning
Introduction and Context
The proliferation of LLMs in clinical question answering has rendered traditional multiple-choice benchmarks effectively saturated, prompting a shift toward open-ended, rubric-based evaluation frameworks that more realistically quantify clinical reasoning in high-stakes contexts. While conversational and holistic metrics in datasets such as HealthBench have demonstrated persistent gaps in LLMs' ability to handle complex clinical decision-making, granular analyses isolating synthesis, prioritization, and safety under contradictory evidence remain sparse and insufficiently explored.
This study addresses these gaps by presenting a clinician-constructed, weighted rubric designed to dissect LLM performance at the atomic inference level. The controlled comparison focuses on three frontier models—GPT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro—across five meticulously crafted clinical scenarios. The scoring pipeline incorporates expert-derived gold answers, MECE (mutually exclusive, collectively exhaustive) decomposed rubric criteria, explicit weighting to reflect clinical priority, and validation with LLM-based auto-raters.
Methodological Framework
Scenario Engineering and Rubric Construction
Five acute or primary care experts authored realistic clinical scenarios involving specialties such as anesthesia, internal/family medicine, emergency medicine, and obstetrics. Each scenario necessitated multi-step reasoning—integrating conflicting signals, managing iatrogenic complications, balancing risk in resource-constrained environments, and synthesizing contradictory diagnostic cues.
For each scenario, the clinician authors composed gold-standard answers from which atomic rubric criteria were extracted. Criteria were weighted from 1 (trivial) to 5 (critical), referencing appropriate guideline sources. The rubrics were subject to clinician peer review for content validity and defensibility, and only tasks for which all three models performed below a 0.60 pass threshold were retained, ensuring upper-tail difficulty.
Evaluation and Quality Assurance Pipeline
Model outputs, generated in a single-turn, no-scaffolding, API-only harness, were independently scored against the constructed rubrics by both human experts and LLM-based autoraters (one for each model family). Where auto-raters diverged from the expert, a reconciliation step was triggered, but final authority rested with the clinician grader. The rubric pass rate for each model was calculated as a weighted ratio, and failure modes were stratified by both criterion weight and clinical category.
This rigorous, reproducible protocol aligns with third-party evaluation guidance, explicitly documenting risks such as contamination, refusals, and reward hacking.
Empirical Findings
Across all scenarios and rubric criteria, Claude achieved the highest mean pass rate (0.47), followed by GPT (0.39) and Gemini (0.37). However, performance exhibited considerable scenario-dependent variance; for the most challenging task (obstetric management of pre-eclampsia with asthma and twins), all models essentially failed (0.07-0.12). This variance underscores the sensitivity of current LLMs to scenario complexity and clinical nuance.
Inversion of Clinical Priority and Critical Failure Modes
A major finding is the inversion between LLM success rate and criterion clinical priority. On criteria weighted as trivial (weight-1), all models excelled (80-90% pass), yet on essential, high-stakes criteria (weight-5), pass rates were only 32.4% (Gemini), 33.3% (GPT), and 41.7% (Claude). None of the three models successfully addressed more than half of the critical requirements, with 52% of all critical criteria universally missed.
Category-Level Analysis
Instruction-following (87.5%) and formatting/style (100%) are effectively solved across all models; by contrast, pass rates for clinical reasoning (28.5–42.3%) and safety (38.8–44.8%) remain suboptimal. This quantitative dissociation points to a vulnerability: models systematically produce outputs with surface fluency and formal correctness but miss decisive, risk-laden inferences and integrative reasoning.
Autorater Calibration
LLM-based autoraters displayed high raw agreement (92.8–94.7%) with clinician expert judgments in binary met/not-met evaluation. While encouraging for the use of LLMs in calibration and scalability, the protocol maintains expert arbitration for final validity, precluding reliance on automated adjudication in clinical settings.
Qualitative Examination of Systematic Failures
Universal critical failures included:
- Omission of synthesis under contradiction (e.g., disregarding improving biomarkers that undermine prior diagnoses).
- Mechanism-level reasoning lapses (failure to connect polypharmacy to renal injury and impaired drug clearance).
- Inadequate risk/benefit analysis (e.g., not appropriately balancing labetalol's relative contraindication in asthma within obstetric hypertension).
- Failing to recommend withholding futile interventions (neglect of non-resuscitation protocols in catastrophic transfer scenarios).
These failures are representative, not exceptional, and reveal the models’ inability to perform chained causal reasoning and context-specific prioritization.
Implications and Future Directions
The study illustrates that atomic, weighted, expert-motivated rubric scoring reveals clinically relevant model deficiencies invisible to holistic or preference-based ratings. Given that high-fidelity formatting and generic instruction following can coexist with dangerous omissions in reasoning and safety, reliance on current LLMs for autonomous clinical decision support remains contraindicated in critical contexts.
From a methodological perspective, the rubric-centric pipeline offers notable advantages for fine-grained supervision, reward modeling, and model alignment. Criteria that discriminate between models, especially at high weights, are natural candidates for targeted contrastive learning or reinforcement learning with rubric-derived rewards, mitigating susceptibility to preference model ambiguities.
Scaling this approach—across hundreds of scenarios and thousands of atomic criteria—could enable statistically robust, specialty-stratified, and cross-systemally transportable evaluation benchmarks. This is essential both for quantifying incremental improvements in frontier models and for identifying "last-mile" reasoning tasks that continue to resist LLM automation.
Conclusion
This controlled evaluation demonstrates that state-of-the-art LLMs systematically fail to satisfy the majority of critical clinical reasoning and safety criteria in complex, clinician-engineered scenarios, despite achieving near-perfect performance on superficial and stylistic tasks. The observed inversion of clinical priority—with models passing unimportant criteria but missing essential ones—demands caution in deployment and underscores the necessity for rubrics that surface decisive inference failures. The authors’ scalable, validity-preserving pipeline lays groundwork for a new generation of high-resolution, clinically meaningful benchmarks and highlights priority areas for future model development and alignment.