Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
Dan Iter Kelvin Guu Google Research Larry Lansing Google Research Dan Jurafsky Computer Science Department, Stanford University
Abstract
Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations. We propose Conpono111Code is available at https://github.com/google-research/language/tree/master/language/conpono and https://github.com/daniter-cu/DiscoEval, an inter-sentence objective for pretraining language models that models discourse coherence and the distance between sentences. Given an anchor sentence, our model is trained to predict the text $k$ sentences away using a sampled-softmax objective where the candidates consist of neighboring sentences and sentences randomly sampled from the corpus. On the discourse representation benchmark DiscoEval, our model improves over the previous state-of-the-art by up to 13% and on average 4% absolute across 7 tasks. Our model is the same size as BERT-Base, but outperforms the much larger BERT-Large model and other more recent approaches that incorporate discourse. We also show that Conpono yields gains of 2%-6% absolute even for tasks that do not explicitly evaluate discourse: textual entailment (RTE), common sense reasoning (COPA)
中文速览
预训练语言模型（如BERT）在词级表示上已经做得很好，但对句子间的篇章连贯性（discourse coherence）关注不足。针对这一问题，研究者提出了名为Conpono的跨句预训练目标：给定一个锚定句，模型需要在大量候选句（包括同段落的难负样本和随机抽取的负样本）中判别出距离锚定句恰好k个句子的目标句，本质上是一个对比学习任务，同时建模了句子间的顺序、位置和连贯性。实验结果显示，在专门评估篇章表示的基准DiscoEval的7个任务上，Conpono平均绝对提升4%、最高提升13%，在参数量仅相当于BERT-Base的情况下超越了更大的BERT-Large及其他专门针对篇章设计的模型；此外，在文本蕴含（RTE）、常识推理（COPA）和阅读理解（ReCoRD）等非篇章专项任务上也有2%–6%的绝对提升。这项工作表明，将篇章连贯性显式纳入预训练目标，能够以较低的额外计算代价为语言模型带来跨任务的普遍收益。
原文 arXiv:2005.10389;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.10389v1