Efficient summarization with read-again and copy mechanism
Wenyuan Zeng†, Wenjie Luo‡, Sanja Fidler‡, Raquel Urtasun‡ †Tsinghua University, ‡University of Toronto {wenjie, fidler,
Abstract
Encoder-decoder models have been widely used to solve sequence to sequence prediction tasks. However current approaches suffer from two shortcomings. First, the encoders compute a representation of each word taking into account only the history of the words it has read so far, yielding suboptimal representations. Second, current decoders utilize large vocabularies in order to minimize the problem of unknown words, resulting in slow decoding times. In this paper we address both shortcomings. Towards this goal, we first introduce a simple mechanism that first reads the input sequence before committing to a representation of each word. Furthermore, we propose a simple copy mechanism that is able to exploit very small vocabularies and handle out-of-vocabulary words. We demonstrate the effectiveness of our approach on the Gigaword dataset and DUC competition outperforming the state-of-the-art.
中文速览
针对文本摘要任务中编码器-解码器模型存在的两大缺陷——编码器只能"向前看"导致词表示不充分，以及解码器为处理生僻词而依赖庞大词表导致推理缓慢——研究者提出了"Read-Again（再读）"机制和配套的复制机制（copy mechanism）。前者让编码器先完整读一遍输入句子，再利用第一遍的全局信息偏置第二遍编码，使每个词的表示都能感知完整上下文；后者则允许解码器在生成摘要时直接从源文本中复制词语，从而把解码词表压缩到极小规模，并能为词表外（OOV）词汇提取更准确的嵌入表示。在 Gigaword 数据集和 DUC 竞赛上的实验表明，该方法超越了当时的最优基准，同时大幅提升了解码速度。这项工作将仿人类"通读后精读"的直觉融入神经网络，为兼顾摘要质量与推理效率提供了一条简洁有效的思路。
原文 arXiv:1611.03382;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.03382v1