Open-Source Tool for Evaluating Human-Generated vs. AI-Generated Medical Notes Using the PDQI-9 Framework

Abstract: Background: The increasing use of AI in healthcare documentation necessitates robust methods for evaluating the quality of AI-generated medical notes compared to those written by humans. This paper introduces an open-source tool, the Human Notes Evaluator, designed to assess clinical note quality and differentiate between human and AI authorship. Methods: The Human Notes Evaluator is a Flask-based web application implemented on Hugging Face Spaces. It employs the Physician Documentation Quality Instrument (PDQI-9), a validated 9-item rubric, to evaluate notes across dimensions such as accuracy, thoroughness, clarity, and more. The tool allows users to upload clinical notes in CSV format and systematically score each note against the PDQI-9 criteria, as well as assess the perceived origin (human, AI, or undetermined). Results: The Human Notes Evaluator provides a user-friendly interface for standardized note assessment. It outputs comprehensive results, including individual PDQI-9 scores for each criterion, origin assessments, and overall quality metrics. Exportable data facilitates comparative analyses between human and AI-generated notes, identification of quality trends, and areas for documentation improvement. The tool is available online at https://huggingface.co/spaces/iyadsultan/human_evaluator . Discussion: This open-source tool offers a valuable resource for researchers, healthcare professionals, and AI developers to rigorously evaluate and compare the quality of medical notes. By leveraging the PDQI-9 framework, it provides a structured and reliable approach to assess clinical documentation, contributing to the responsible integration of AI in healthcare. The tool's availability on Hugging Face promotes accessibility and collaborative development in the field of AI-driven medical documentation.