DisSent: Learning Sentence Representations from Explicit Discourse Relations
Allen Nie 1 Erin D. Bennett†† Noah D. Goodman1,2 1Department of Computer Science 2Department of Psychology Stanford University equal contribution
Abstract
Learning effective representations of sentences is one of the core missions of natural language understanding. Existing models either train on a vast amount of text, or require costly, manually curated sentence relation datasets. We show that with dependency parsing and rule-based rubrics, we can curate a high quality sentence relation task by leveraging explicit discourse relations. We show that our curated dataset provides an excellent signal for learning vector representations of sentence meaning, representing relations that can only be determined when the meanings of two sentences are combined. We demonstrate that the automatically curated corpus allows a bidirectional LSTM sentence encoder to yield high quality sentence embeddings and can serve as a supervised fine-tuning dataset for larger models such as BERT. Our fixed sentence embeddings achieve high performance on a variety of transfer tasks, including SentEval, and we achieve state-of-the-art results on Penn Discourse Treebank’s implicit relation prediction task.
中文速览
想让机器真正理解句子的语义，既不想标注海量数据，又不想只靠预测词语这类低层次信号——DisSent就是为了解决这个两难困境而提出的。研究者发现，自然语言中大量存在"因为""但是""所以"这类话语标记词（discourse markers），它们天然标注了两个句子之间的深层语义关系；于是他们用依存句法分析（dependency parsing）自动从书籍语料中抽取出470余万对句子，构建了一个无需人工标注却高质量的训练集，并训练模型预测连接两句话的是哪个话语标记词。基于这一任务，双向LSTM句子编码器学到了质量可观的句子向量，同时把该数据集用于微调BERT也带来了额外增益。最终，DisSent在多项迁移学习基准（SentEval）上达到与InferSent相当甚至更优的表现，并在宾州话语树库（Penn Discourse Treebank）隐式关系预测任务上刷新了当时的最优结果——这说明利用语言中自然存在的话语结构，是一条低成本、高效率学习句子语义表示的可行路径。
原文 arXiv:1710.04334;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.04334v4