Large Language Models Encode Clinical Knowledge
Karan Singhal Google Research, Shekoofeh Azizi Google Research, Tao Tu Google Research, S. Sara Mahdavi Google Research, Jason Wei Google Research, Hyung Won Chung Google Research, Nathan Scales Google Research, Ajay Tanwani Google Research, Heather Cole-Lewis Google Research, Stephen Pfohl Google Research, Perry Payne Google Research, Martin Seneviratne Google Research, Paul Gamble Google Research, Chris Kelly Google Research, Nathaneal Schärli Google Research, Aakanksha Chowdhery Google Research, Philip Mansfield Google Research, Blaise Agüera y Arcas Google Research, Dale Webster Google Research, Greg S. Corrado Google Research, Yossi Matias Google Research, Katherine Chou Google Research, Juraj Gottweis Google Research, Nenad Tomasev DeepMind Yun Liu Google Research, Alvin Rajkomar Google Research, Joelle Barral Google Research, Christopher Semturs Google Research, Alan Karthikesalingam Google Research, Vivek Natarajan Google Research,
Abstract
Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but the quality bar for medical and clinical applications is high. Today, attempts to assess models’ clinical knowledge typically rely on automated evaluations on limited benchmarks. There is no standard to evaluate model predictions and reasoning across a breadth of tasks. To address this, we present MultiMedQA, a benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries; and HealthSearchQA, a new free-response dataset of medical questions searched online. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias.
中文速览
医学领域的AI应用门槛极高，但现有评估体系既零散又粗糙，根本无法全面衡量大语言模型（LLM）在临床场景中的真实水平。为此，研究团队构建了MultiMedQA这一综合基准，整合了六个已有医学问答数据集并新增了消费者常见健康搜索问题集HealthSearchQA，同时设计了一套涵盖事实准确性、潜在危害、偏见等多维度的人工评估框架。他们将5400亿参数的PaLM及其指令微调版Flan-PaLM在该基准上进行测试，结合少样本、思维链等提示策略，Flan-PaLM在美国执照医师考试题（MedQA）上达到67.6%的准确率，比此前最优结果高出逾17%；进一步通过"指令提示微调"（instruction prompt tuning）这一高效对齐方法打造的Med-PaLM，在临床医生人工评审中将"符合科学共识"的回答比例从61.9%提升至92.6%，潜在有害回答比例也从29.7%降至5.8%，已接近真实临床医生水平。这项工作既展示了大语言模型在医学领域的应用潜力，也清晰揭示了当前模型在公平性、偏见等方面仍存在的关键缺口，为未来安全可靠的临床AI系统的开发提供了重要的评估基准与方法参考。
原文 arXiv:2212.13138;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.13138v1