MedCalc-Bench: Medical Calculator Benchmark

• MedCalc-Bench is a large-scale benchmark designed to evaluate LLMs on clinically relevant quantitative reasoning tasks using real and synthetic patient notes.
• It systematically integrates equation-based calculations and rule-based scoring from 55 MDCalc-derived calculators with physician-verified step-by-step explanations.
• Evaluation protocols using zero-shot and chain-of-thought methods reveal model performance differences and spotlight challenges in label fidelity and clinical alignment.

MedCalc-Bench is a large-scale, publicly available benchmark specifically constructed to evaluate and train LLMs on clinically relevant medical calculation tasks. Unlike traditional medical QA benchmarks, MedCalc-Bench is focused on end-to-end quantitative and rule-based reasoning as encountered in real-world evidence-based decision support, with the goal of assessing the extent to which LLMs can read de-identified clinical narratives, extract relevant attributes, and apply standardized medical calculators as a physician would on platforms such as MDCalc.com (Khandekar et al., 2024, Ye et al., 22 Dec 2025, Mao et al., 31 Oct 2025).

1. Definition, Scope, and Structure

MedCalc-Bench was introduced by Khandekar et al. as the first dataset designed to systematically assess the ability of LLMs to solve authentic medical calculator tasks. The benchmark comprises 1,047 problem instances, each defined by: (a) a de-identified patient note (real or synthetic), (b) a calculator-specific free-response question, (c) the ground-truth numerical or categorical answer, and (d) a step-by-step physician-verified explanation of the calculation or scoring. These tasks span 55 distinct calculators popularized via MDCalc.com, selected to capture commonly used clinical formulas and scoring systems (Khandekar et al., 2024, Ye et al., 22 Dec 2025).

MedCalc-Bench decomposes the calculator tasks as follows:

• Equation-based calculators: Tasks requiring numerical evaluation using explicit parametric formulas (e.g., Cockcroft-Gault creatinine clearance, body mass index, Du Bois body surface area).
• Rule-based scores: Tasks defined by categorical aggregation of clinical features according to published rubrics (e.g., Wells’ criteria for DVT, Apgar Score, Glasgow Coma Scale, CHA₂DS₂-VASc stroke-risk).

Patient notes were sourced primarily from Open-Patients (≈180,000 EHR-style notes), supplemented with synthetic or clinician-authored cases to ensure comprehensive coverage across task types.

2. Examples of Calculator Tasks and Representative Formulas

MedCalc-Bench operationalizes each calculator with explicit variable lists, standardized formulas, and Python implementations for label generation. Key examples include:

• Body Mass Index (BMI):

$$\mathrm{BMI} = \frac{\mathrm{weight\;(kg)}}{[\mathrm{height\;(m)}]^2}$$

• Cockcroft–Gault Creatinine Clearance:

$$\mathrm{CrCl} = \frac{(140 - \mathrm{age}) \times \mathrm{weight\;(kg)}}{72 \times \mathrm{serum\;creatinine\;(mg/dL)}} \times \begin{cases} 1.00 & \text{(male)} \ 0.85 & \text{(female)} \end{cases}$$

• Wells’ Criteria for DVT (rule-based): Sum of sub-scores for criteria such as active cancer, immobilization, leg swelling, pitting edema, and alternative diagnoses, with point assignments as per published guidelines.

This structured approach ensures that each MedCalc-Bench instance requires the model to extract relevant entities from free text and perform multi-step reasoning to obtain the correct outcome (Khandekar et al., 2024, Mao et al., 31 Oct 2025).

3. Evaluation Protocols and Model Performance

Benchmarked models include both open-source and proprietary LLMs (Llama 3, Mistral, GPT-3.5, GPT-4), as well as healthcare-tuned variants (PMC-LLaMA, MediTron). Three primary prompting strategies are used:

• Zero-Shot Direct: Model produces only the final answer.
• Zero-Shot Chain-of-Thought (CoT): Model explains its reasoning stepwise before answering.
• One-Shot CoT: As CoT but with an in-context example for the target calculator.

Models are scored using strict rules: rule-based and date outputs require exact match; numerical equation-based outputs permit ±5% tolerance. The principal metric is accuracy (fraction correct by these criteria).

Model/Prompt	Accuracy (%) (One-Shot CoT)
GPT-4	50.9
Llama 3 70B	39.5
Mistral-7B	~1.5 (zero-shot); ~49 (finetuned)

Accuracy varies by calculator type: physical measures (e.g., BMI) are easiest (77.5% for GPT-4), while complex rule-based severity indices present the greatest challenge (27.5%). No model reaches levels suitable for unassisted clinical deployment (Khandekar et al., 2024).

4. Label Quality and Physician-in-the-Loop Auditing

Subsequent research highlights critical issues in ground-truth label fidelity within MedCalc-Bench. The original gold labels were generated via a two-stage LLM- and script-based pipeline: feature extraction from notes (via GPT-4) followed by rule-based aggregation in Python per calculator. Systematic audits revealed that between one-quarter and one-third of test labels diverge from physician judgment due to three main error classes:

• Feature-extraction errors: GPT-4 misread or hallucinated clinical features.
• Aggregation logic mismatches: Implementation bugs in calculator scripts.
• Clinical ambiguity: Cases unscorable due to missing, out-of-scope, or ambiguous information.

A scalable audit pipeline employing advanced agentic verifiers (e.g., Gemini 2.5 Pro with code tools) triaged instances for correction. Independent relabeling and targeted physician adjudication reduced the symmetric mean absolute percentage error (sMAPE) from 72.7% to 20.1% and raised physician agreement from 20% to 74% on a validation sample (Ye et al., 22 Dec 2025).

5. Impact on Downstream RL Fine-Tuning and Model Alignment

MedCalc-Bench plays a critical role as a reward reference for reinforcement learning (RL)-based LLM alignment. Label noise in gold answers was shown to compromise model performance: controlled experiments under Group Relative Policy Optimization (GRPO) demonstrated that using physician-aligned labels (vs. initial labels) for Qwen3-8B fine-tuning yields an absolute test-set accuracy gain of +8.7% (71.4% vs. 62.6%). This performance uplift is on par with the improvement from moving zero-shot to few-shot settings on prior large models (Ye et al., 22 Dec 2025).

A plausible implication is that benchmark maintenance and high-fidelity labeling are essential to avoid model misalignment, spurious learning, and biased downstream evaluation in safety-critical medical AI settings.

6. Limitations and Critiques of MedCalc-Bench

Several structural limitations of MedCalc-Bench have been identified:

• Domain breadth: Only 55 calculators, with sparse representation outside a few core specialties (nephrology, cardiology, nutrition).
• Complexity: Most tasks involve single algebraic formulas or simple rule aggregation—rarely requiring nested logic, multi-conditional selection, or unit conversions.
• Contextual rigor: Few tasks embed patient information in free-form clinical note style, so information extraction remains relatively constrained.
• Label fidelity: As shown, a substantial fraction of original benchmarks encode extraction or logic-derived noise, necessitating periodic audits for ongoing validity (Mao et al., 31 Oct 2025, Ye et al., 22 Dec 2025).

This analysis motivated the creation of MedCalc-Eval, which expands the number of calculators (709 vs. 55), domains, calculation types (629 formula-based, 80 scale-based), and introduces higher-order logic, nested formulas, multi-condition logic, and more realistic note-to-calculation pipelines (Mao et al., 31 Oct 2025).

7. Implications for Benchmark Design and Future Directions

MedCalc-Bench established the paradigm for quantitative, calculator-based clinical LLM evaluation and training. Its rapid uptake and impact illustrate the importance of faithfully reproducing real-world physician workflows in AI benchmarks. However, findings from subsequent audits underscore that in safety-critical applications, benchmarks must be treated as "living documents": subject to routine re-evaluation as model capabilities and clinical practice evolve.

Best practices identified include:

• Regular, automated benchmarking triage using advanced LLM verifiers to prioritize clinician review.
• Transparent documentation and versioning of label updates, including explicit "NA" handling when scorable ground truth is undefined.
• Prioritization of expert time on the most contentious and ambiguous instances to maximize annotation ROI (Ye et al., 22 Dec 2025).

MedCalc-Bench remains an influential testbed, but its limitations have informed the development of more comprehensive resources such as MedCalc-Eval and corresponding RL environments (MedCalc-Env), marking continued evolution in the field of medical AI benchmarking (Khandekar et al., 2024, Mao et al., 31 Oct 2025, Ye et al., 22 Dec 2025).