Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Harsha Nori*‡ Yin Tat Lee* Sheng Zhang* Dean Carignan Richard Edgar Nicolo Fusi Nicholas King Jonathan Larson Yuanzhi Li Weishung Liu Renqian Luo Scott Mayer McKinney† Robert Osazuwa Ness Hoifung Poon Tao Qin Naoto Usuyama Chris White Eric Horvitz‡
Abstract
Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities without intensive training of models with specialty knowledge. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of the specialist capabilities of GPT-4 on medical challenge benchmarks in the absence of special training. In distinction to the intentional use of simple prompting to highlight the model’s out-of-the-box capabilities, we perform a systematic exploration of prompt engineering to boost performance. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical question-answering datasets. The prompt engineering methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. As a culmination o
中文速览
大型通用语言模型GPT-4在医疗专业考题上的表现，究竟能否在不做任何专科微调的情况下超越专门训练的医疗模型？研究团队为此设计了一套名为Medprompt的提示工程（prompt engineering）方法，核心做法是让GPT-4自己为训练题生成思维链推理步骤（chain-of-thought），再动态检索相似例题构成少样本示范，最后通过多次采样投票集成答案——整个流程无需任何医学专家参与标注。在MultiMedQA包含的全部9个医疗问答基准数据集上，Medprompt均刷新了最优成绩，在美国执照医师资格考试题库MedQA上首次突破90%准确率，错误率比此前最强专科模型Med-PaLM 2降低了27%，且调用模型的次数减少了约一个数量级。更重要的是，这套方法完全不含医学专属设计，在电气工程、法律、会计、护理等多个专业领域的能力测试上同样大幅提升了GPT-4的表现，说明通用基础模型只需巧妙引导就能激发出深层专科能力，而无需昂贵的领域微调。
原文 arXiv:2311.16452;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.16452v1