MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes

Abstract: Doctor agents are moving beyond single-turn answer generation toward evolving clinical decision systems. Within an outpatient episode, they acquire evidence, use examination and consultation resources, and decide when to finalize a diagnosis and management plan. Across episodes, their behavior may change through memory, retrieval, reflection, or other update mechanisms. Current evaluations only partially cover this setting. Fixed-input medical QA benchmarks score final answers from complete inputs, whereas many interactive benchmarks still focus on individual encounters or fixed runs, providing limited support for evaluating how episode-level decisions interact with cross-episode experience. We introduce MedEvoEval, an executable longitudinal evaluation framework based on action-gated simulated outpatient episodes. Each source case is converted into role-specific patient, examination, and manager views; evidence is revealed only through valid actions; and each episode records a structured trace that links observations, actions, final outputs, manager scores, and optional experience write-back. We release a runnable E&D artifact with 700 processed episodes, provenance notes, schemas, an episode runner, scoring scripts, configurations, example logs, analysis code, and trajectory- and step-level derivatives. Experiments show that episode traces expose process costs hidden by final-answer scoring, show how MDT-style consultation reallocates resources, and support longitudinal analyses of memory maturation, held-out transfer, update-stage response, and backward retention. Together, these results show that MedEvoEval provides a concrete basis for evaluating whether doctor agents improve through experience, transfer useful behavior, and retain earlier capabilities over time.

• The paper introduces MedEvoEval, a simulation-based framework that assesses evolving doctor agents across multiple clinical episodes.
• The paper implements a detailed evaluation protocol capturing action validation, process costs, and memory-driven adaptation using specific metrics like ΔS and BWT.
• The paper demonstrates that memory maturation significantly enhances clinical performance and resource efficiency while maintaining stability against forgetting.

MedEvoEval: A Framework for Longitudinal Evaluation of Continually-Evolving Doctor Agents

Motivation and Problem Statement

The emergence of interactive, memory-augmented LLM-based doctor agents necessitates evaluation protocols that extend beyond static, final-answer accuracy measures. Conventional medical QA benchmarks and most interactive agent environments fail to capture dynamic evidence acquisition, resource use, and behavior adaptation driven by cross-episode experience. MedEvoEval systematically formulates this multi-level evaluation problem, providing both within- and cross-episode assessment of doctor agents in a controlled, reproducible simulation environment.

Figure 1: Overview of MedEvoEval. Source cases are decomposed into role-bound episode views, enabling episode execution, process-level trace and output scoring, memory write-back, and longitudinal assessment.

Framework Design

MedEvoEval formalizes doctor-agent evaluation as a sequence of simulated outpatient episodes, using role-specific representations (patient, examination, manager) and a tightly controlled protocol for evidence revelation, permitted actions, and process logging. Central evaluation primitives include action-gated evidence acquisition, structured process traces, rubric-based scoring, and controlled experience write-back with memory for cross-episode adaptation.

Key design attributes:

• Role-specific observability: Agents only access information unlocked by valid ASK or REQUEST_TEST actions.
• Action validation and process cost measurement: Invalid actions, excessive or redundant examinations, and premature decisions are logged and penalized.
• Memory as an ablation variable: Episodes run with or without replayable memory cards encoding successful or failed experience; retrieval and memory conditions are precisely configured for controlled experiments.
• Longitudinal structure: The episode corpus enables both within-stream memory maturation analysis and true held-out transfer to previously unseen cases.

  Figure 2: MedEvoEval experiment overview captures process trace analysis, MDT-style (multidisciplinary team) consultation, longitudinal memory maturation, transfer, and update-retention diagnostics.

Evaluation Protocol and Metrics

MedEvoEval introduces composite quantitative metrics that comprehensively capture clinical output quality, process cost, and experiential adaptation:

• Outcome metrics: Rubric-weighted combination of diagnosis, evidence, and management quality ($S$), strict diagnosis accuracy, and thresholded pass rate.
• Process metrics: Number of interactive turns ($T$), examinations ($E$), and invalid test rates ($R$) derived from the event log.
• Utility metric: Aggregates outcome and process cost via a fixed-weight formula.

Longitudinal and continual learning metrics:

• Stage-wise learning gain: Progression across staged episode windows quantifies memory maturation.
• External transfer: Comparison between mature-memory (H3) and no-memory (H0) performance on held-out cases.
• Backward transfer (BWT): Captures retention and backward degradation following updates.

Empirical Findings

Process Trace Adds Cost Awareness

Comparative experiments across four API-accessible LLM doctor agents show that answer-only scoring obscures clinically relevant distinctions in resource usage and error modes. The event trace reveals that similar answer accuracy can mask large variabilities in inefficiencies or invalid interactions.

Figure 3: Model comparisons show distinctions in output quality (diagnosis, evidence, plan), process (turns, test requests), score distributions, and decoupling of thoroughness from excess examination.

MDT Consultations Shift Resource Profiles

MDT-style consultation (with same or stronger consultants) primarily adjusts process allocation: it reduces examination requests but increases consultation turns, leaving total diagnostic accuracy mostly unchanged. This decoupling of resource allocation from final outcome facilitates fine-grained policy analysis for collaborative agent strategies.

Figure 4: MDT experiment decomposes outcome, utility, process cost, diagnosis-change, and consultant behavior.

Longitudinal Adaptation: Memory Maturation and Transfer

Empirical analysis on a 500-stage longitudinal stream establishes three primary claims:

1. Memory requires maturation: Initial memory (E1) provides minimal benefit, but repeated experience accumulation (E2, E3) yields significant gains in outcome score (e.g., $S$ increases from 0.246 at E0 to 0.430 at E3) and reduces average interactive turns.
2. Held-out transfer is positive: Mature memory improves unseen case performance ($\Delta S_{H3-H0} = +0.075$), demonstrating genuine experiential generalization, not just stream replay.
3. Stability against forgetting: Backward transfer measurements ($\mathrm{BWT}_S = -0.0087$) confirm only mild backward degradation, indicating that update-driven adaptation does not catastrophically overwrite prior knowledge.

   Figure 5: Stage-level learning curves showing total score gains, utility improvement, accuracy, pass rate, turn, and test usage under progressive memory conditions.

   

   Figure 6: External transfer comparison demonstrates that mature memory (H3) enhances performance and efficiency versus no memory (H0) on held-out cases.

   

   Figure 7: Update and retention diagnostics split adaptation from backward retention, quantifying the effect of update intervals and measuring BWT.

Mechanism Ablations

Further experiments vary the type and retrieval breadth of memory to ablate mechanism effects. Success-plus-failure memory cards have similar average impact to failure-only in small subsets, while retrieval breadth ($K_{\mathrm{ret}}$) tunes the balance between decision quality and brevity in episode execution.

Figure 8: Memory ablations probing the effect of memory content and retrieval scale on outcome, utility, and strict accuracy.

Limitations and Implications

MedEvoEval is a simulation protocol: patient and exam responses are scripted, and management scoring is rubric-based (though supported by a single-physician audit). The framework's explicit memory and process controls permit rigorous, reproducible evaluations but do not guarantee real-world diagnostic safety or readiness. Further, the reported baseline experiments use a limited set of API-restricted LLMs; broad-scale or open-source model evaluation is left for future work.

Despite these bounds, MedEvoEval's separation of clinical outcome, process cost, memory-driven generalization, and stability provides a more complete basis for assessing evolving doctor agents. Practical implications include:

• Facilitating transparent comparisons of continual learning mechanisms in medical agent settings.
• Enabling fine-grained ablation and policy analysis for collaborative/multimodal agent systems (e.g., MDT evaluation).
• Supporting compliance and auditability via structured logs of all evidence exposure and action sequences.
• Informing real-time deployment risk monitoring by separating answer correctness from process glitches and inefficiencies.

Conclusion

MedEvoEval provides a technically rigorous, executable framework for evaluating continually adapting doctor agents as longitudinal decision systems. By decomposing process, memory, transfer, and stability in a unified protocol, it addresses the shortcomings of prior QA- and single-episode-oriented benchmarks. The released artifact and empirical analyses offer a platform for further research on continual learning, resource-constrained planning, and safe longitudinal evolution of AI-driven medical agents (2606.28900).