Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

Abstract: Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training. Rather than using simple prompting to highlight the model's out-of-the-box capabilities, we perform a systematic exploration of prompt engineering. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks. The prompting methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, based on a composition of several prompting strategies. With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite. The method outperforms leading specialist models such as Med-PaLM 2 by a significant margin with an order of magnitude fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27% reduction in error rate on the MedQA dataset over the best methods to date achieved with specialist models and surpasses a score of 90% for the first time. Beyond medical problems, we show the power of Medprompt to generalize to other domains and provide evidence for the broad applicability of the approach via studies of the strategy on exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.

• The paper demonstrates that generalist AI models like GPT-4 can achieve state-of-the-art performance on medical question-answering benchmarks through advanced prompt engineering.
• It introduces Medprompt, a novel strategy that selects similar training examples and employs shuffle ensembling to enhance logical reasoning steps.
• The study highlights the cross-domain potential of such techniques while urging caution due to risks of hallucination and real-world applicability challenges.

In the field of artificial intelligence, there is a growing interest in whether AI models that are not specifically trained for a particular domain can perform as well as, or even outperform, those that are. A notable area of this investigation is the medical domain, where reliable and accurate information is critical.

A recent study has shown that generalist models like GPT-4 can indeed exhibit expertise in specialized domains like medicine through innovative "prompt engineering." Prompt engineering is a method of carefully designing queries to elicit the most knowledgeable and specific responses from AI models.

GPT-4 has been indicated to achieve state-of-the-art results in medical question-answering benchmarks. This performance entails answering medical exam-style questions, akin to those used in the Medical Licensing Examination, at remarkably high accuracy rates. These results are surprising because, traditionally, it was assumed that models like GPT-4 would not demonstrate specialist capabilities without either being fine-tuned on domain-specific data or being given prompts created by experts in the domain.

A novel prompting strategy, called Medprompt, was introduced in the study. Here's how it works: First, the model identifies and uses training examples that are most similar to the task at hand. Then, it uses these examples to generate reasoning steps logically leading up to an answer. These generated reasoning steps have proven to be very effective, even more so than those crafted by human experts in medicine. Finally, the model's answers are bolstered by shuffle ensembling - a method that shuffle the answer choices to ensure consistent outputs.

The remarkable aspect of the study's findings is not only GPT-4's high performance in the medical field but also that the used prompting methods are general and not specifically designed for the medical domain. Further tests demonstrated that similar methods could be applied to multiple domains, including engineering, law, philosophy, and nursing, showing improvements over basic model querying methods.

Despite its groundbreaking results, the research emphasizes caution. Indeed, strong benchmark performances on massive, indiscriminately-sourced data may not directly translate to real-world applicability or accuracy. There also remains the risk of the AI model providing incorrect information or 'hallucinations,' which might be amplified by better prompting strategies – a critical consideration especially in high-stakes fields such as medicine.

In conclusion, the study presents an intriguing insight into the capabilities of generalist AI models. By using sophisticated prompting strategies, GPT-4's latent specialty knowledge can be unlocked, potentially reducing or even eliminating the need for expensive domain-specific model training or the manual crafting of prompts by specialists. The methodology has broad implications for the application of AI across various fields, not just in the field of medicine. However, the implementation of such technology requires careful consideration and thorough examination of its limitations, especially in contexts where accuracy is a matter of utmost concern.