Dynamic Diagnostic Evaluation Paradigm

• Diagnostic Evaluation Paradigm is a methodological framework that quantifies and interprets diagnostic reasoning through iterative, process-aware assessments.
• It employs interactive multi-turn dialogues, contamination-resistant case generation, and multi-level metrics to capture both outcome and process quality.
• The paradigm integrates accuracy, efficiency, and rubric-based evaluations to expose diagnostic reasoning gaps and drive improvements in clinical AI.

A diagnostic evaluation paradigm is a methodological framework designed to assess, quantify, and interpret the reasoning, accuracy, and process quality of diagnostic agents—whether human, algorithmic, or hybrid—across clinical, educational, or technical domains. In recent clinical AI, the paradigm has shifted from static, single-turn accuracy metrics to process-aware, dynamic, and multi-level assessment protocols that capture the complexity and uncertainty inherent in real-world diagnostic problem solving. This article provides an in-depth overview of contemporary diagnostic evaluation paradigms, synthesizing advances in dynamic medical LLM evaluation, benchmarking, interactive simulation, process quality assessment, and multi-metric integration, with reference to the architecture and findings of recent frameworks such as ClinDEF (Tang et al., 29 Dec 2025).

1. Dynamic Diagnostic Evaluation: Motivation and Foundational Principles

Traditional diagnostic benchmarks rely on static, single-turn question-answering, using metrics like accuracy or top-k recall. However, such approaches fail to emulate the clinical diagnostic process, which is fundamentally an iterative, path-dependent sequence of information gathering, hypothesis generation, evidence collection, and hypothesis refinement. This gap motivates dynamic evaluation paradigms, characterized by:

• Process fidelity: The diagnostic process is modeled as a multi-turn, interactive dialogue, recapitulating how real clinicians iteratively gather information, order examinations, and revise their hypotheses.
• Contamination resistance: Dynamic case generation eliminates memorization and data leakage by generating novel cases at evaluation time.
• Multi-level granularity: Assessment captures not just final outcomes (accuracy) but process-level metrics such as efficiency, evidence integration, and safety-critical reasoning steps.

Frameworks such as ClinDEF instantiate these principles through knowledge-grounded, consistently generated cases, simulated multi-agent dialogue, and multi-dimensional, rubric-based assessment protocols (Tang et al., 29 Dec 2025).

2. Framework Architectures: Systemic Components and Case Generation

Contemporary dynamic evaluation frameworks typically comprise three core modules:

• Case generator: Synthesizes patient profiles in a structured and contamination-resistant way. For example, ClinDEF leverages a curated disease–symptom knowledge graph ($K_G$) combined with a narrative knowledge base ($K_E$). A generative LLM ($\Theta$) enforces graph-derived consistency constraints, yielding clinically coherent case profiles $(d, T_d, P_\text{info}, S_d)$.
• Multi-agent diagnostic simulator: Orchestrates interactive diagnostic dialogue. Distinct roles include the doctor agent ($A_D$, the model under evaluation), patient agent ($A_P$, deterministic and symptom-grounded), and examiner agent ($A_E$, deterministic and test-grounded).
• Process and outcome evaluator: Computes outcome metrics (accuracy), process metrics (efficiency), and rubric-based diagnostic quality scores, often inspired by clinical assessment rubrics (e.g., OSCEs).

Dynamic frameworks, by generating cases algorithmically under ontological constraints and enforcing strict segregation of training and evaluation instances, provide robust contamination controls and broader clinical coverage (Tang et al., 29 Dec 2025).

3. Interactive Diagnostic Dialogue: Protocol and Decision Space

The heart of the dynamic paradigm is an interactive, multi-turn diagnostic protocol. The dialogue system comprises:

• Dialogue state $H_t$: Partial history encoding all utterances up to turn $t$.
• Doctor’s action space: At each turn, the doctor agent selects among “Ask” (subjective query), “Test” (order objective examination), or “Diag” (commit to diagnosis).
• Turn-by-turn evolution: Actions condition on the full dialogue history, enforcing a Markov property wherein each decision is informed by the evidence accumulated so far.

The protocol isolates and quantifies canonical reasoning operations:

• Information gathering (targeted questioning)
• Evidence synthesis (integration of test findings)
• Hypothesis management (revision and narrowing down of differentials)

Dialogue continues until the agent commits to a diagnosis or reaches a maximum allowed number of turns. This abstraction enables reproducibility, comparability across models, and interpretable error analysis (Tang et al., 29 Dec 2025).

4. Multi-Dimensional Assessment: Metrics and Rubric-Based Scoring

Dynamic paradigms adopt multi-dimensional evaluation structured explicitly across outcome, process, and quality axes:

A. Diagnostic Accuracy

$$\text{Acc}_i = \mathbf{1}(\text{diagnosis}_i = d_i)$$

where $d_i$ is the ground-truth and $\text{diagnosis}_i$ the agent’s final answer.

B. Efficiency Metrics

• $\text{TotTurns}_i$: Number of dialogue turns to diagnosis.
• $\text{PosFinds}_i / \text{NegFinds}_i$: Number of positive/negative findings.
• Positive Hit Rate ($\text{PHR}_i$):

$$\mathrm{PHR}_i = \frac{\text{PosFinds}_i}{\text{PosFinds}_i + \text{NegFinds}_i}$$

• Aggregate efficiency score:

$E_i = \frac{\text{Acc}_i}{\text{TotTurns}_i}$

C. Rubric-Based Diagnostic Quality (DQS)

$\mathrm{DQS}_i = \sum_{d \in D} w_d \cdot S_{i,d}$

where $D$ is the set of detailed axes (CCE: chief complaint exploration, HC: history completeness, ECI: evidence chain integrity, TJ: test justification, DDx: differential diagnosis breadth, DC: diagnostic correctness, DU: uncertainty management), $w_d$ are weights, and $S_{i,d}$ are sub-scores per axis.

This rubric-based assessment, inspired by OSCEs, quantifies dimensions such as initial exploration depth, evidence use, logical reasoning, breadth of differentials, correctness, and risk management. Efficiency and rubric scores are analyzed within each clinical-reasoning phase (symptom elicitation, hypothesis generation, conclusion/safety), yielding process-level insight distinct from outcome-focused static benchmarks (Tang et al., 29 Dec 2025).

5. Experimental Design and Empirical Findings

Dynamic evaluation frameworks such as ClinDEF deploy multi-split, contamination-controlled benchmarks in large-scale experiments:

• Dataset generation: Typically multiple independent test sets, each with hundreds of algorithmically generated clinical cases.
• Simulator validation: Physician review for diagnosis leakage (e.g., >99% of dialogues leak-free) and clinical coherence.
• Model comparison: Closed- and open-source LLMs are evaluated with unified protocol settings (temperature, top-p, maximum turns).
• Results interpretation:
  • State-of-the-art models reach diagnostic accuracy below 75%, with notable efficiency trade-offs (i.e., models with higher PHR diagnose in fewer turns).
  • Maximum observable diagnostic quality score (DQS) reaches ~70/100, with systematic weaknesses in initial information gathering and uncertainty management.
  • Fine-grained analysis discriminates models by their questioning efficiency, logical coherence, and safety-aware reasoning, exposing limitations not discoverable by raw accuracy alone.

These empirical insights validate the paradigm’s ability to reveal failure modes and process deficiencies invisible to static, single-turn assessments (Tang et al., 29 Dec 2025).

6. Process-Aware Evaluation: Implications and Broader Significance

Dynamic diagnostic evaluation paradigms represent a major methodological advance:

• Holistic assessment: They unify outcome, process, and quality metrics into a single evaluation protocol, capturing the complexity of diagnostic reasoning.
• Generalizability and contamination resistance: Automated case generation under strict ontological and procedural controls enables scalable, contamination-minimized benchmarking across clinical domains.
• Comparative granularity: Individual reasoning phases (information gathering, hypothesis revision, conclusion) receive targeted, rubric-driven analysis, enabling fine-stratified performance comparisons and targeted model refinement.
• Limitation exposure: By simulating real diagnostic workflows, dynamic paradigms expose systematic gaps (e.g., in uncertainty handling, breadth of differential, evidence justification) that are masked by traditional static Q&A metrics.

Such architectures and protocols set a new standard for both rigorous scientific benchmarking and for regulatory validation of diagnostic AI systems (Tang et al., 29 Dec 2025).

7. Positioning Within the Broader Evaluation Landscape

The diagnostic evaluation paradigm described herein is situated among a suite of dynamic, process-sensitive assessment frameworks:

Framework	Protocol Type	Case Generation	Assessment Granularity	Key Metrics
ClinDEF	Simulated dialogue	Knowledge-graph + LLM	Multi-level, rubric	Accuracy, efficiency, DQS
DiagnosisArena	Static, open-ended	Expert-segmented real cases	Outcome-only	Top-k accuracy
MSDiagnosis	Multi-step, self-check	EMR retrieval + ICL	Forward/backward/reflection/refinement	Entity F1, Macro-Recall
H-DDx	List-form DDx, hierarchical	Real vignettes	Taxonomy-aware	HDF1
LLM-Mini-CEX	Simulator-guided	Patient simulator	Rubric via binary items	% pass per axis

ClinDEF’s paradigm is distinguished by fully dynamic case synthesis, rigorous multi-agent dialogue, and rubric-based, phase-resolved process QA. This framework exposes clinical reasoning gaps and supports model iteration and regulatory validation with process fidelity unmatched by static or partially dynamic benchmarks (Tang et al., 29 Dec 2025).

---

References

• "ClinDEF: A Dynamic Evaluation Framework for LLMs in Clinical Reasoning" (Tang et al., 29 Dec 2025)
• "DiagnosisArena: Benchmarking Diagnostic Reasoning for LLMs" (Zhu et al., 20 May 2025)
• "MSDiagnosis: A Benchmark for Evaluating LLMs in Multi-Step Clinical Diagnosis" (Hou et al., 2024)
• "H-DDx: A Hierarchical Evaluation Framework for Differential Diagnosis" (Lim et al., 4 Oct 2025)
• "LLM-Mini-CEX: Automatic Evaluation of LLM for Diagnostic Conversation" (Shi et al., 2023)