BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language
Abstract
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models Peters et al. (2018a); Radford et al. (2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications.
中文速览
如何让机器真正"双向"理解一句话的上下文，一直是自然语言处理（NLP）的核心难题——此前主流模型要么只从左往右读文本，要么只是把两个单向模型粗糙地拼在一起。BERT（基于Transformer的双向编码器表示）通过两个无监督预训练任务来解决这个问题：一是"遮蔽语言模型"（Masked Language Model），随机遮住句子中15%的词让模型去猜，从而迫使模型同时利用左右两侧的上下文；二是"下一句预测"（Next Sentence Prediction），让模型判断两个句子是否真实相邻，以学习句间关系。预训练完成后，只需在顶部加一个简单的输出层并用少量标注数据微调，BERT就能适配问答、文本推断、命名实体识别等各类任务，在11项NLP基准测试上全面刷新纪录，例如GLUE综合得分提升7.7个百分点、SQuAD问答F1值达到93.2。BERT的意义在于证明了"深度双向预训练+轻量微调"这一范式的强大威力，极大降低了为每个任务单独设计复杂模型的门槛，成为此后NLP领域预训练大模型路线的重要基石。
原文 arXiv:1810.04805;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.04805v2