Enhancing Clinical Concept Extraction with Contextual Embeddings
Yuqi Si Jingqi Wang Hua Xu Kirk Roberts School of Biomedical Informatics The University of Texas Health Science Center at Houston
Abstract
Neural network-based representations (``embeddings") have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (e.g., ELMo, BERT) have further pushed the state-of-the-art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText). Both off-the-shelf, open-domain embeddings and pretrained clinical embeddings from MIMIC-III are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings, and compare these on four concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pre-training time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextua
中文速览
用于从电子病历中自动识别医学概念（如疾病、检查、治疗）的临床NLP模型，长期依赖word2vec、GloVe等"静态词向量"，但这类方法无法区分同一词在不同语境下的含义。研究团队系统比较了五种词表示方法——word2vec、GloVe、fastText、ELMo和BERT——在四个标准临床概念抽取数据集（i2b2 2010/2012、SemEval 2014/2015）上的表现，同时对比了通用领域预训练模型与在MIMIC-III近200万份真实病历上重新预训练的临床专属模型。结果显示，在MIMIC-III上预训练的BERT（BERTLARGE）在全部四个任务上均刷新了最优成绩，F1分别达到90.25、93.18、80.74和81.65，且上下文嵌入（contextual embeddings）能捕捉传统词向量完全忽略的语义歧义信息。这项工作为临床NLP社区提供了明确的最佳实践：当计算资源允许时，应优先选用在大规模临床文本上预训练的上下文嵌入模型，而非沿用静态词向量。
原文 arXiv:1902.08691;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.08691v4