Semi-supervised sequence tagging with bidirectional language models
Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, Russell Power Allen Institute for Artificial Intelligence
Abstract
Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks. However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data. In this paper, we demonstrate a general semi-supervised approach for adding pre-trained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks. We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.
中文速览
命名实体识别（NER）和句法块分析（Chunking）等序列标注任务依赖双向循环神经网络来捕捉词语的上下文信息，但这类网络只能在有限的标注数据上训练，难以充分学习语言规律。研究者提出了一种半监督方法——TagLM，核心思路是先在海量无标注文本上预训练双向语言模型（bidirectional language model），再把语言模型为每个词生成的上下文向量（LM embedding）直接拼接到序列标注模型的中间层，让标注模型"借用"语言模型所学到的丰富语义和句法知识。在CoNLL 2003 NER任务上，加入语言模型嵌入后F₁从90.87%提升到91.93%，在CoNLL 2000 Chunking任务上达到96.37% F₁，两项均刷新当时最优记录，甚至超过了那些额外使用人工标注数据或领域词典的系统。这项工作意义在于，它证明了无需任何额外标注资源，仅靠大规模无监督语言模型就能为下游序列标注任务带来显著且稳定的提升，为后来ELMo、BERT等预训练语言表示方法奠定了重要基础。
原文 arXiv:1705.00108;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.00108v1