BioMegatron: Larger Biomedical Domain Language Model
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, Raghav Mani NVIDIA / Santa Clara, California, USA
Abstract
There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books. Yet, most works do not study the factors affecting each domain language application deeply. Additionally, the study of model size on domain-specific models has been mostly missing. We empirically study and evaluate several factors that can affect performance on domain language applications, such as the sub-word vocabulary set, model size, pre-training corpus, and domain transfer. We show consistent improvements on benchmarks with our larger BioMegatron model trained on a larger domain corpus, contributing to our understanding of domain language model applications. We demonstrate noticeable improvements over the previous state-of-the-art (SOTA) on standard biomedical NLP benchmarks of named entity recognition, relation extraction, and question answering. Model checkpoints and code are available at ngc.nvidia.com and github.com/NVIDIA/NeMo.
中文速览
大规模预训练语言模型迁移到生物医学领域已被广泛探索，但影响其性能的关键因素——尤其是模型规模——此前研究得并不深入。研究者基于Megatron-LM框架构建了BioMegatron，系统比较了子词词表（subword vocabulary）、模型参数量（345M到1.2B）、预训练语料和领域迁移策略对命名实体识别（NER）、关系抽取（RE）和问答（QA）三类任务的影响。实验发现，词表设计对NER的影响远比模型规模更关键——能把生物医学术语保留为完整词元的领域词表可显著降低实体被切碎的概率，从而提升识别准确率；而更大的模型在QA任务上有所增益，但收益在345M之后趋于平缓；从通用领域模型出发进行充分的领域继续预训练，最终可以媲美从头训练的纯领域模型。这项工作揭示了"通吃型"大模型并不能在所有领域任务上都胜过针对性设计的模型，为后续面向特定领域和任务的语言模型开发提供了清晰的方法论参考。
原文 arXiv:2010.06060;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.06060v2