Towards a Personal Health Large Language Model

Abstract: In health, most LLM research has focused on clinical tasks. However, mobile and wearable devices, which are rarely integrated into such tasks, provide rich, longitudinal data for personal health monitoring. Here we present Personal Health LLM (PH-LLM), fine-tuned from Gemini for understanding and reasoning over numerical time-series personal health data. We created and curated three datasets that test 1) production of personalized insights and recommendations from sleep patterns, physical activity, and physiological responses, 2) expert domain knowledge, and 3) prediction of self-reported sleep outcomes. For the first task we designed 857 case studies in collaboration with domain experts to assess real-world scenarios in sleep and fitness. Through comprehensive evaluation of domain-specific rubrics, we observed that Gemini Ultra 1.0 and PH-LLM are not statistically different from expert performance in fitness and, while experts remain superior for sleep, fine-tuning PH-LLM provided significant improvements in using relevant domain knowledge and personalizing information for sleep insights. We evaluated PH-LLM domain knowledge using multiple choice sleep medicine and fitness examinations. PH-LLM achieved 79% on sleep and 88% on fitness, exceeding average scores from a sample of human experts. Finally, we trained PH-LLM to predict self-reported sleep quality outcomes from textual and multimodal encoding representations of wearable data, and demonstrate that multimodal encoding is required to match performance of specialized discriminative models. Although further development and evaluation are necessary in the safety-critical personal health domain, these results demonstrate both the broad knowledge and capabilities of Gemini models and the benefit of contextualizing physiological data for personal health applications as done with PH-LLM.

• The paper introduces PH-LLM, a model fine-tuned from Gemini for personal health that achieves near-expert performance.
• PH-LLM integrates textual and sensor data using over 857 domain-specific case studies to generate personalized recommendations.
• The model outperforms experts with 79% and 88% scores on sleep and fitness exams, underpinning its relevance for health monitoring.

Personal Health LLM (PH-LLM)

The paper "Towards a Personal Health LLM" presents PH-LLM, a LLM fine-tuned from Gemini, targeting personal health and wellness applications. The model was evaluated across three core tasks: generating personalized insights and recommendations in sleep and fitness, answering expert-level multiple-choice questions, and predicting patient-reported sleep outcomes from sensor data.

Model Architecture and Training

PH-LLM is adapted from Gemini Ultra 1.0, which has shown high accuracy across professional health exams. The fine-tuning process incorporated a mixture of domain-specific training datasets, including over 857 sleep and fitness case studies designed with expert input. These were split into training, validation, and test sets to benchmark model performance. PH-LLM utilizes a multimodal approach, integrating textual and numerical health data, such as sensor readings from wearables, to produce contextually rich outputs.

Case Study Creation and Evaluation

The case studies were generated from real-world scenarios, capturing diverse physiological and behavioral data over extended periods. Sleep studies focused on metrics like bedtime consistency and the balance of sleep stages, while fitness studies assessed metrics such as cardiovascular training loads and subjective readiness. Expert human evaluations revealed that PH-LLM's outputs were closely aligned with those of domain experts, particularly in the fitness domain (Figure 1).

Figure 1: Case Study Human Evaluation Results. Mean ratings given by experts for the case study subsections across sleep and fitness domains.

Performance on Professional Exams

PH-LLM achieved impressive results on domain-specific exams, scoring 79% on sleep medicine and 88% on fitness multiple-choice questions, outperforming average human expert scores. This performance indicates high model capacity in handling domain-specific knowledge and reasoning, a critical benchmark for deploying LLMs in health-related fields (Figure 2).

Figure 2: Overall performance of professional exams across PH-LLM, different Gemini models, and Med-PaLM 2.

Multimodal Integration for Patient-Reported Outcomes

The paper demonstrated PH-LLM’s ability to predict patient-reported outcomes (PROs) by combining wearable sensor data with self-reported assessments. The model used an MLP adapter with a multimodal token encoder to map sensor data into the model’s context, enabling it to predict outcomes like sleep disturbances and impairment efficiently. This approach outperformed zero and few-shot prompting methods in predicting PROs, closely mirroring the capabilities of specialized discriminative models (Figure 3).

Figure 3: Performance of PH-LLM measured via AUROC.

Implications and Future Directions

The research suggests that LLMs can be effectively tailored for personal health monitoring by interpreting longitudinal sensor data, thus supporting personalized health interventions. The model's ability to predict subjective wellbeing offers significant potential for integration into user interfaces, facilitating health tracking and recommendations. Future research should aim to refine the accuracy further, ensure safety in recommendations, and explore the model's adaptability to real-time health data streams.

Conclusion

The PH-LLM showcases how LLMs can transcend traditional boundaries of data interpretation in personal health contexts, achieving near-expert level performance on specialized tasks. With ongoing developments, the integration of LLMs into personal health tools could revolutionize patient education, monitoring, and wellness planning. Such models stand to greatly enhance the precision and personalization of health recommendations, making them a crucial component of digital health innovations.