Almanac Copilot: Towards Autonomous Electronic Health Record Navigation

Abstract: Clinicians spend large amounts of time on clinical documentation, and inefficiencies impact quality of care and increase clinician burnout. Despite the promise of electronic medical records (EMR), the transition from paper-based records has been negatively associated with clinician wellness, in part due to poor user experience, increased burden of documentation, and alert fatigue. In this study, we present Almanac Copilot, an autonomous agent capable of assisting clinicians with EMR-specific tasks such as information retrieval and order placement. On EHR-QA, a synthetic evaluation dataset of 300 common EHR queries based on real patient data, Almanac Copilot obtains a successful task completion rate of 74% (n = 221 tasks) with a mean score of 2.45 over 3 (95% CI:2.34-2.56). By automating routine tasks and streamlining the documentation process, our findings highlight the significant potential of autonomous agents to mitigate the cognitive load imposed on clinicians by current EMR systems.

• The paper introduces Almanac Copilot, an autonomous agent that streamlines EHR navigation using a 33 billion parameter LLM and advanced embedding techniques.
• It employs a multi-component architecture integrating FHIR functions, clinical calculators, and an EHR-QA dataset to achieve a 74% task completion rate.
• The study addresses clinical challenges such as alert fatigue and excessive data entry, paving the way for more efficient and autonomous healthcare systems.

Almanac Copilot: Towards Autonomous Electronic Health Record Navigation

Introduction

The paper "Almanac Copilot: Towards Autonomous Electronic Health Record Navigation" introduces Almanac Copilot, an autonomous agent designed to address inefficiencies in electronic health record (EHR) navigation, which have been linked to clinician burnout and compromised quality of care. The paper provides an overview of the architecture and evaluation of Almanac Copilot, an autonomous EHR agent capable of performing tasks such as information retrieval, data manipulation, and alert prioritization.

Architecture

Almanac Copilot utilizes a multi-component architecture to streamline the clinical workflow. Key components include:

• LLM: This is a 33 billion parameter instruction-tuned transformer decoder. The LLM is optimized for computational efficiency on consumer-grade hardware, leveraging advancements like Multi-Query Attention (MQA), Rotary Positional Embeddings (RoPE), and RMSNorm for stable training.
• Embedding Model: Matryoshka Representation Learning (MRL) ensures information embedding at varying granularity levels, adapting the model for diverse healthcare computing and latency requirements.
• Tools: Almanac Copilot integrates several tools accessible through pre-defined functions, facilitating complex tasks that require cross-system reasoning and decision-making. These include FHIR-based EHR functions, medical literature search engines, clinical calculators, and robust database queries.

  Figure 1: Overview of the Almanac Copilot Architecture. Upon receiving a query, the system dynamically selects a subset of APIs from a predetermined list of functions (i.e., FHIR functions, browser, calculator), optimizing the process to meet the specific requirements of the query.

The comprehensive architecture allows the autonomous agent to efficiently handle myriad tasks while ensuring compatibility with modern EHR systems through adherence to the FHIR standard.

Methods

EHR-QA Dataset

An evaluation benchmark named EHR-QA is created to simulate common clinician workflows. This consists of 300 synthetic EHR-facing questions designed to mimic tasks frequently encountered in clinical settings. The dataset is generated using template questions with substitutions made based on a randomly selected patient's history from the proxy EHR data.

Performance Evaluation

Using the EHR-QA dataset, Almanac Copilot achieves a significant task completion rate of 74%. Evaluation criteria include function choice, parameter selection, and script validity, assessing the ability to properly automate EHR tasks. Comparisons against other systems such as ChatGPT-4, Claude 3 Opus, and BioMistral demonstrate competitive performance, highlighting its efficacy in routine clinical operations.

Figure 2: Performance Evaluation of Almanac Copilot, ChatGPT-4, Claude 3 Opus, and BioMistral on EHR-QA. The stacked bar plot and heatmaps illustrate the frequency and score obtained across 300 synthetic questions.

Discussion

Almanac Copilot delivers an impactful contribution towards reducing the cognitive load imposed by contemporary EMRs, functioning as a Level 1 autonomous agent. It addresses issues of alert fatigue, excessive data entry, and poor usability by automating routine tasks. However, challenges remain in mitigating hallucinations that arise from incomplete data inputs, affecting task accuracy. Future improvements could include fine-tuning techniques like reinforcement learning and broadening the dataset to capture diverse clinical queries beyond its current scope.

Efforts to advance towards Level 2 autonomy include enhancing the agent’s contextual understanding, reducing latency, and integrating multimodal capabilities for comprehensive healthcare delivery. Such advancements promise real-world deployment that aligns closely with clinician workflows and healthcare delivery efficiencies.

Conclusion

The introduction of Almanac Copilot marks a significant step towards alleviating the burdens inherent in clinician interactions with EHR systems. By capitalizing on the strengths of contemporary AI architectures and integrating practical healthcare standards, it offers promising solutions to long-standing issues affecting clinician productivity and wellbeing. Continued development and refinement of autonomous health record systems could lead to far-reaching improvements in healthcare efficiency and clinician satisfaction.