DiscoFuse: A Large-Scale Dataset for Discourse-Based Sentence Fusion
Mor Geva Tel Aviv University、Eric Malmi Google AI \ANDIdan Szpektor Google AI、Jonathan Berant Tel Aviv University, Allen Institute for AI Work done during internship at Google AI. Work done at Google AI.
Abstract
Sentence fusion is the task of joining several independent sentences into a single coherent text. Current datasets for sentence fusion are small and insufficient for training modern neural models. In this paper, we propose a method for automatically-generating fusion examples from raw text and present DiscoFuse, a large scale dataset for discourse-based sentence fusion. We author a set of rules for identifying a diverse set of discourse phenomena in raw text, and decomposing the text into two independent sentences. We apply our approach on two document collections: Wikipedia and Sports articles, yielding 60 million fusion examples annotated with discourse information required to reconstruct the fused text. We develop a sequence-to-sequence model on DiscoFuse and thoroughly analyze its strengths and weaknesses with respect to the various discourse phenomena, using both automatic as well as human evaluation. Finally, we conduct transfer learning experiments with WebSplit, a recent dataset for text simplification. We show that pretraining on DiscoFuse substantially improves performance on WebSplit when viewed as a sentence fusion task.
中文速览
句子融合（sentence fusion）的任务是把若干独立句子合并成一段连贯的文本，但现有标注数据集规模极小，根本不够训练当代神经网络模型。为此，作者提出了一套从原始语料库自动生成融合样本的规则体系，通过识别话语现象（如话语连接词、回指/前指代词、关系从句、同位语等）并将完整句子"拆解"为两个独立句，反向构造出训练对，最终在维基百科和体育文章两个语料上生成了共6000万条样本，形成大规模数据集DiscoFuse。他们在DiscoFuse上训练了基于Transformer的序列到序列模型，发现模型在处理结构性融合（如同位语、关系从句）时表现较好，但在选择正确话语连接词和代词回指时仍存在明显弱点，众包评估也验证了数据集约85%的样本质量可靠。进一步的迁移学习实验表明，用DiscoFuse预训练后再在WebSplit等小规模数据集上微调，性能显著提升，说明这一超大规模数据集对推动句子融合乃至更广泛的文本生成研究具有重要价值。
原文 arXiv:1902.10526;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.10526v3