Capabilities of GPT-4 on Medical Challenge Problems
Harsha Nori Microsoft Nicholas King Microsoft Scott Mayer McKinney OpenAI Dean Carignan Microsoft Eric Horvitz Microsoft
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4 [Ope23], a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the United States Medical Licensing Examination (USMLE), a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study calibration of the probabilities, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points
中文速览
现役最强通用大模型GPT-4在医学执照考试（美国医师执照考试，USMLE）上到底能考几分？研究团队购买了官方真题，系统评测了GPT-4在USMLE三个阶段以及多个公开医学问答基准（MultiMedQA）上的表现。结果显示，GPT-4在不做任何医学专项微调、不使用复杂提示技巧的情况下，平均得分超过86%，比及格线高出逾20个百分点，也全面碾压了专门针对医学知识进行训练的Med-PaLM等模型。研究还发现GPT-4的概率校准（calibration）——即模型对自己答对概率的预估准确度——显著优于前代GPT-3.5，这在医疗这类高风险场景中至关重要。这意味着通用大模型已具备超越医学专科模型的临床推理能力，对医学教育、考试辅助乃至未来的临床辅助决策都具有重要参考价值，但研究者同时强调准确性与安全性仍需谨慎对待。
原文 arXiv:2303.13375;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.13375v2