Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again
Bernal Jiménez Gutiérrez1, Nikolas McNeal1, Clay Washington1, You Chen2, Lang Li1, Huan Sun1, Yu Su1 1The Ohio State University, 2Vanderbilt University
Abstract
Large pre-trained language models (PLMs) such as GPT-3 have shown strong in-context learning capabilities, which are highly appealing for domains such as biomedicine that feature high and diverse demands of language technologies but also high data annotation costs. In this paper, we present the first systematic and comprehensive study to compare the few-shot performance of GPT-3 in-context learning with fine-tuning smaller (i.e., BERT-sized) PLMs on two representative biomedical information extraction (IE) tasks: named entity recognition and relation extraction. We follow the true few-shot setting Perez et al. (2021) to avoid overestimating models’ few-shot performance by model selection over a large validation set. We also optimize GPT-3’s performance with known techniques such as contextual calibration and dynamic in-context example retrieval. However, our results show that GPT-3 still significantly underperforms compared to simply fine-tuning a smaller PLM. In addition, GPT-3 in-context learning also yields smaller gains in accuracy when more training data becomes available. More in-depth analyses further reveal issues of in-context learning that may be detrimental to IE tasks i
中文速览
生物医学文本中充斥着大量需要被结构化提取的信息，而标注数据既昂贵又稀缺，这让许多研究者将目光投向GPT-3的"上下文学习"（in-context learning）——只需提供少量示例便可完成任务，无需更新模型参数。本文对GPT-3上下文学习与微调小型预训练语言模型（如BERT）在生物医学命名实体识别和关系抽取任务上的少样本性能进行了首个系统性比较，采用严格的"真实少样本"评估设置，并为GPT-3引入了动态示例检索、上下文校准等多种优化技术。然而结果表明，即便如此，GPT-3的表现仍显著落后于仅用100条标注样本微调的小型模型，且随着训练数据增加，其性能提升也更为有限；深入分析还发现GPT-3在处理"无实体"或"无关系"的负样本时存在明显弱点。这一发现对生物医学从业者具有重要的实践价值：在更好的上下文学习方法出现之前，微调小型领域专用语言模型仍是成本更低、效果更稳定的现实选择。
原文 arXiv:2203.08410;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.08410v3