How many patients could we save with LLM priors?

Abstract: Imagine a world where clinical trials need far fewer patients to achieve the same statistical power, thanks to the knowledge encoded in LLMs. We present a novel framework for hierarchical Bayesian modeling of adverse events in multi-center clinical trials, leveraging LLM-informed prior distributions. Unlike data augmentation approaches that generate synthetic data points, our methodology directly obtains parametric priors from the model. Our approach systematically elicits informative priors for hyperparameters in hierarchical Bayesian models using a pre-trained LLM, enabling the incorporation of external clinical expertise directly into Bayesian safety modeling. Through comprehensive temperature sensitivity analysis and rigorous cross-validation on real-world clinical trial data, we demonstrate that LLM-derived priors consistently improve predictive performance compared to traditional meta-analytical approaches. This methodology paves the way for more efficient and expert-informed clinical trial design, enabling substantial reductions in the number of patients required to achieve robust safety assessment and with the potential to transform drug safety monitoring and regulatory decision making.

• The paper introduces LLM-informed priors to enhance Bayesian safety models in clinical trials, reducing required sample sizes while preserving statistical power.
• Utilizing both blind and disease-informed prompting strategies, the study demonstrates improved predictive performance over traditional meta-analytical priors.
• Empirical evaluations using NSCLC trial data reveal that models achieved comparable accuracy with only 80% of training data, indicating significant cost and patient benefits.

How Many Patients Could We Save with LLM Priors?

Introduction

The paper "How many patients could we save with LLM priors?" (2509.04250) investigates the application of LLMs for Bayesian prior elicitation in hierarchical Bayesian modeling of adverse events (AEs) in clinical trials. The study explores the potential of using LLM-informed priors to reduce the number of patients required in clinical trials while maintaining statistical power. This is achieved by leveraging pre-trained LLMs to directly obtain informative priors, thus incorporating clinical expertise into Bayesian safety models.

Hierarchical Bayesian AE Modeling with LLM-Informed Priors

Model Formulation

The modeling framework utilizes individual patient data (IPD) from multi-center clinical trials. The hierarchical Bayesian model structures patient AE counts as Poisson-distributed with site-specific rates following a Gamma distribution. Key model components include:

1. Patient-level likelihood: $y_{ij} \sim \text{Poisson}(\lambda_j)$
2. Site-specific AE rates: $\lambda_j \sim \text{Gamma}(\alpha, \beta)$
3. Hyperpriors: $\alpha \sim \pi_\alpha(\cdot)$ and $\beta \sim \pi_\beta(\cdot)$

The novel contribution is the use of LLM-derived priors for the hyperparameters $\alpha$ and $\beta$, enabling integration of domain expertise into the model.

LLM-Based Prior Elicitation Strategy

The paper systematically explores two prompting strategies for leveraging LLMs:

1. Blind Prompting: General queries posed to LLMs without disease-specific context.
2. Disease-Informed Prompting: Contextual queries providing specific clinical and disease-related information.

The study utilizes Llama 3.3, a general-purpose LLM, and MedGemma, a model fine-tuned for biomedical contexts.

Empirical Evaluation

The methodology was empirically evaluated using real-world clinical trial data from a non-small cell lung cancer (NSCLC) study. The evaluation focused on predictive performance improvement and sample size reduction compared to traditional meta-analytical priors.

Cross-Validation

The cross-validation experiments demonstrated that LLM-derived priors, particularly with Llama 3.3 using Blind prompts at $T=1.0$, consistently outperformed traditional approaches in Log Predictive Density (LPD), signaling improved predictive performance.

Figure 1: Boxplot of elicited alpha prior parameters for Llama 3.3 across prompt types and temperatures.

Figure 2: Boxplot of elicited beta prior parameters for Llama 3.3 across prompt types and temperatures.

Sample Efficiency Analysis

The study quantitatively assessed the sample efficiency of LLM-derived priors. Significant reductions in required sample sizes were observed, with models achieving comparable predictive performance using only 80% of the training data.

Discussion

The results suggest that LLM-based prior elicitation can lead to substantial improvements in both predictive accuracy and sample efficiency. This has far-reaching implications for clinical trial design, potentially reducing costs and patient burden while enhancing statistical robustness.

The prior parameter variability analysis highlights potential tuning mechanisms through prompt strategies and temperature settings that can be employed to control model behavior. The broader applicability of such methods across different clinical contexts remains an area for future exploration.

Future Directions

Future research could focus on validation across a broader spectrum of diseases, extend the approach to other types of clinical endpoints, and explore integration with regulatory frameworks. The synergistic use of LLM-based data augmentation alongside prior elicitation also presents a promising avenue for further enhancement of clinical trial analyses.

Conclusion

This study illustrates the capacity of LLM-informed priors to enhance the statistical power and efficiency of clinical trials through improved AE predictive modeling. By translating expert clinical knowledge into quantifiable Bayesian priors, the approach paves the way for more informed and efficient clinical trial designs, ultimately facilitating accelerated and cost-effective drug development.