Zero-Shot Medical Text De-Identification

• Zero-shot de-identification is an automatic method for anonymizing clinical texts by removing protected health information without task-specific training.
• The approach employs techniques like prompt engineering for LLMs and local NLP pipelines, balancing scalability with regulatory requirements.
• Performance metrics such as recall and F1 score guide compliance, with local solutions often surpassing cloud APIs in meeting HIPAA and GDPR standards.

Zero-shot medical text de-identification refers to the automatic removal or masking of protected health information (PHI) from unstructured clinical narratives, performed by systems that do not require task-specific supervised fine-tuning or annotated examples for training. In this setting, models—typically LLMs or pretrained NLP pipelines—are evaluated on their ability to accurately and comprehensively detect and obfuscate diverse PHI categories directly “out-of-the-box” or given only prompts, making the approach attractive for rapid deployment and scalability in privacy-critical healthcare environments. Zero-shot approaches are assessed on stringent regulatory criteria, including recall and F1 thresholds needed for HIPAA or GDPR expert determination.

1. Zero-Shot De-Identification Architectures and Methodologies

Zero-shot de-identification methods can be broadly categorized into API-driven commercial solutions and prompt-based LLM frameworks. Four primary approaches have been systematically benchmarked:

• Cloud Black-Box APIs: Azure Health Data Services, AWS Comprehend Medical, and OpenAI GPT-4o—all remotely hosted endpoints where no customization or local model adaptation is permitted. Zero-shot operation entails using the base model settings, sometimes with prompt engineering (GPT-4o), but without system message customization, user prompts, or temperature manipulation (except as noted in GPT-4o setups) (Kocaman et al., 21 Mar 2025).
• Local Pretrained Pipelines: John Snow Labs Healthcare NLP (Spark NLP), deployed as a fixed-cost on-premise library with a prebuilt de-identification pipeline, requiring no LLM back-end and minimal code for pipeline loading.

Recent prompt-based methods, exemplified by DeID-GPT (Liu et al., 2023), use GPT-4 with explicitly engineered textual directives that enumerate PHI category definitions, along with concrete examples, sent in a single prompt. No few-shot learning or supervised fine-tuning is involved. The de-identification process comprises:

1. Prompt construction, mapping HIPAA identifiers to PHI entities in the target style.
2. Direct zero-shot inference, where system and user roles are set (in chat form), and the output replaces all matching categories with standardized tokens (e.g., “[redacted]”), thus preserving note structure.

Prompt engineering is essential for maximizing recall and preventing model confusion; best practices include lead-off task statements, explicit replacement directives, concise category-wise rules, and counter-example cycles to minimize prompt ambiguity. In DeID-GPT, temperature is set to 0 to reduce variability, and prompt length is tightly controlled to avoid system focus loss (Liu et al., 2023).

2. Datasets, Annotation Protocols, and PHI Categories

Zero-shot de-identification solutions are evaluated using clinical corpora annotated for PHI. Two key benchmark data sources are reported:

• Clinical Expert-Labeled Set (Kocaman et al., 21 Mar 2025): 48 documents, 45,172 PHI entities, six core PHI types (IDNUM, LOCATION, DATE, AGE, NAME, CONTACT). Annotation was executed with an initial DL model-based pre-annotation, followed by two rounds of medical-domain expert review and dynamic annotation guides. Two levels of granularity are used:
  • Entity-level: full match, partial match, not matched.
  • Token-level: PHI/non-PHI whitespace token tagging.
• 2014 i2b2/UTHealth Corpus (Liu et al., 2023): 1,304 notes, >1,000 test entities, with 7 mapped PHI categories from the original 18 HIPAA “Safe Harbor” identifiers. Gold labels are hand-annotated spans.

Mapping between source PHI schemas and target label sets is resolved upfront, either manually or by semantic similarity computation (the latter can be performed via LLM itself).

3. Performance Metrics and Regulatory Thresholds

Evaluation consistently measures per-category and overall performance at both the entity and token levels. The primary metrics are:

• Precision: $P = \frac{TP}{TP+FP}$
• Recall: $R = \frac{TP}{TP+FN}$
• F1-score: $F_1 = 2\frac{P \cdot R}{P+R}$

Where TP, FP, and FN are counts of true positives, false positives, and false negatives, respectively, either for entity span matches or token boundaries. Regulatory-grade de-identification is defined as F1 ≥ 95% and recall ≥ 95% for PHI detection—criteria reflecting HIPAA expert determination thresholds and standard NIST benchmarks (Kocaman et al., 21 Mar 2025).

Table: Comparative Zero-Shot De-Identification Accuracy (PHI Detection F1)

Method	Entity-level F1	Token-level F1
Healthcare NLP (JSL)	0.96	0.98
Azure Health Data Services	0.91	0.95
AWS Comprehend Medical	0.83	0.91
OpenAI GPT-4o	0.79	0.89
DeID-GPT (GPT-4) [i2b2]	0.99	n/a
ClinicalBERT-NER (tuned)	0.97	n/a

JSL Healthcare NLP is the only solution to exceed the regulatory-grade threshold across all categories and match or surpass human annotator accuracy (≈93–94% F1). GPT-4 in the DeID-GPT study achieves F1 ≈ 0.99 on the i2b2 benchmark with an optimized explicit prompt (Kocaman et al., 21 Mar 2025, Liu et al., 2023).

4. Detailed Workflows: GPT-Based and Traditional Pipelines

GPT-4 (DeID-GPT) Workflow (Liu et al., 2023):

1. Map HIPAA identifiers to PHI categories.
2. Assemble a prompt: task statement, single “replace…” command, concise category rules with examples.
3. Send the prompt and note to GPT-4 API (temperature=0).
4. Model generates a de-identified note with PHI strings replaced by “[redacted]”.

Commercial API/Pretrained NLP Pipeline Workflows (Kocaman et al., 21 Mar 2025):

• Azure and AWS: Synchronous HTTPS POST calls, zero prompt engineering, fixed endpoints for PHI tagging.
• GPT-4o: One-shot demonstration in JSON output appended to the system prompt, default settings for all model parameters except temperature (set to 1.0 for PHI type coverage), post-processing to align outputs to gold labels.
• John Snow Labs Spark NLP: Pipeline load and execution with two lines of code, no remote server or LLM API dependency, customizable de-identification stages.

5. Limitations, Risks, and Deployment Considerations

Cloud vs. Local Deployment:

• Cloud APIs (Azure, AWS, OpenAI) incur per-request or per-token costs, data residency risks, and HIPAA Business Associate Agreement (BAA) limitations (Kocaman et al., 21 Mar 2025, Liu et al., 2023).
• Offline, fixed-cost solutions (JSL Healthcare NLP) bypass cloud-specific regulatory and cost issues, and allow full pipeline control and horizontal scaling.

Failure Modes:

• GPT-4 and similar black-box LLMs can miss entities under non-standard formatting, over-redact numeric tokens, or lose accuracy with excessive prompt length (Liu et al., 2023).
• In cloud LLMs, FN rate failures (especially recall <95%) pose substantial regulatory compliance risks (Kocaman et al., 21 Mar 2025).
• OpenAI GPT-4o, even with tailored prompts, did not reach regulatory-grade performance (F1=0.79 entity-level, 0.89 token-level).

Cost and Scalability:

• For large-scale de-identification (e.g., 1M notes of 5,250 characters each): fixed-cost local pipelines are estimated to be >80% less expensive than cloud options (e.g., $2,418 vs.$13k–$21k), with predictable horizontal scalability (Kocaman et al., 21 Mar 2025).

Compliance and Privacy Recommendations (Liu et al., 2023):

• Avoid cloud LLM use for PHI unless BAA and data residency are confirmed.
• Prefer local/on-premise LLM deployment (e.g., quantized open-source models).
• Implement human-in-the-loop auditing and differential privacy techniques where possible.

6. Comparative Analyses and Implications

Explicit prompt engineering in modern LLMs substantially improves zero-shot de-identification accuracy over default chat completions or black-box API endpoints. DeID-GPT demonstrates near-perfect recall and precision on i2b2 PHI benchmarks when an explicit, rules-based prompt is used, outperforming both untuned and many tuned transformer-based NER models (Liu et al., 2023). However, in real-world, regulatory settings, only local optimized pipelines achieve the recall and F1 scores required for compliance across all categories. This suggests that, although LLMs with explicit prompting are highly promising for rapid deployment or corpus-level anonymization, for high-throughput, regulatory-grade clinical operations, prebuilt local NLP solutions remain critical.

A plausible implication is that ongoing research into local, fine-tunable LLMs (with access control, quantization, and domain-specific adaptation) may eventually converge with the regulatory performance currently seen with deterministic, offline pipelines, provided challenges in false-negative reduction and reliable auditability are addressed. As of the latest reported results, local pre-trained clinical NLP pipelines remain the only consistently regulatory-grade and economically scalable solution for zero-shot medical text de-identification (Kocaman et al., 21 Mar 2025).