SEAL: Segment-wise Extractive-Abstractive Long-form Text Summarization
Yao Zhao Google Research, Brain Team、Mohammad Saleh Google Research, Brain Team、Peter J. Liu Google Research, Brain Team
Abstract
Most prior work in the sequence-to-sequence paradigm focused on datasets with input sequence lengths in the hundreds of tokens due to the computational constraints of common RNN and Transformer architectures. In this paper, we study long-form abstractive text summarization, a sequence-to-sequence setting with input sequence lengths up to 100,000 tokens and output sequence lengths up to 768 tokens. We propose SEAL, a Transformer-based model, featuring a new encoder-decoder attention that dynamically extracts/selects input snippets to sparsely attend to for each output segment. Using only the original documents and summaries, we derive proxy labels that provide weak supervision for extractive layers simultaneously with regular supervision from abstractive summaries. The SEAL model achieves state-of-the-art results on existing long-form summarization tasks, and outperforms strong baseline models on a new dataset/task we introduce, Search2Wiki, with much longer input text. Since content selection is explicit in the SEAL model, a desirable side effect is that the selection can be inspected for enhanced interpretability.
中文速览
针对超长文本（最多10万个词元）的自动摘要难题，现有方案往往因为Transformer的平方复杂度而不得不将输入截断，导致大量关键信息丢失。研究者提出了SEAL模型，其核心思路是在编码器-解码器的注意力机制中加入一个"选段打分器"：先把超长输入切分成若干片段分别编码，再由打分器动态挑选与当前解码片段最相关的输入段落来做注意力计算，从而实现稀疏、高效的长文本处理；训练时同时使用摘要监督信号和由输入-摘要文本相似度自动生成的弱监督抽取标签，不需要额外预训练。在arXiv、PubMed等已有长文本摘要基准上，SEAL达到了当时最优水平，并在作者新建的Search2Wiki数据集（用搜索结果自动生成完整维基百科词条，输入长度可达10万词元）上显著超越各对比基线。这项工作的意义在于，它提供了一个可扩展到超长输入的端到端单模型框架，同时由于选段过程可见，模型的可解释性也得到了改善。
原文 arXiv:2006.10213;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.10213v1