Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
Xiaochuang Han Jacob Eisenstein Georgia Institute of Technology XH is now at Carnegie Mellon University and JE is now at Google Research. Some of the work was performed while JE was visiting Facebook AI Research.
Abstract
Contextualized word embeddings such as ELMo and BERT provide a foundation for strong performance across a wide range of natural language processing tasks by pretraining on large corpora of unlabeled text. However, the applicability of this approach is unknown when the target domain varies substantially from the pretraining corpus. We are specifically interested in the scenario in which labeled data is available in only a canonical source domain such as newstext, and the target domain is distinct from both the labeled and pretraining texts. To address this scenario, we propose domain-adaptive fine-tuning, in which the contextualized embeddings are adapted by masked language modeling on text from the target domain. We test this approach on sequence labeling in two challenging domains: Early Modern English and Twitter. Both domains differ substantially from existing pretraining corpora, and domain-adaptive fine-tuning yields substantial improvements over strong BERT baselines, with particularly impressive results on out-of-vocabulary words. We conclude that domain-adaptive fine-tuning offers a simple and effective approach for the unsupervised adaptation of sequence labeling to diffic
中文速览
预训练语言模型（如BERT）在新闻、维基百科等现代文本上表现出色，但当目标领域与预训练语料库差异悬殊时（例如莎士比亚时代的早期现代英语或Twitter社交媒体文本），其效果会大打折扣。为此，研究者提出了"领域自适应微调"（domain-adaptive fine-tuning）方法，即先用目标领域的无标注文本对BERT进行掩码语言模型训练，再用源域有标注数据完成序列标注任务的微调，整个过程无需任何目标域标注数据。实验表明，该方法在词性标注和命名实体识别任务上均显著优于直接迁移的BERT基线，尤其对词表外词汇（out-of-vocabulary words）的错误率降低超过50%，且在适应新域的同时不损害原域性能。这一结果说明，对于历史文献处理、社交媒体分析等偏门领域，只需少量额外的无监督训练即可大幅提升现成预训练模型的适用性，具有很强的实用价值。
原文 arXiv:1904.02817;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.02817v2