Incorporating Copying Mechanism in Sequence-to-Sequence Learning
Jiatao Gu† Zhengdong Lu‡ Hang Li‡ Victor O.K. Li† †Department of Electrical and Electronic Engineering, The University of Hong Kong {jiataogu, ‡Huawei Noah’s Ark Lab, Hong Kong {lu.zhengdong,
Abstract
We address an important problem in sequence-to-sequence (Seq2Seq) learning referred to as copying, in which certain segments in the input sequence are selectively replicated in the output sequence. A similar phenomenon is observable in human language communication. For example, humans tend to repeat entity names or even long phrases in conversation. The challenge with regard to copying in Seq2Seq is that new machinery is needed to decide when to perform the operation. In this paper, we incorporate copying into neural network-based Seq2Seq learning and propose a new model called CopyNet with encoder-decoder structure. CopyNet can nicely integrate the regular way of word generation in the decoder with the new copying mechanism which can choose sub-sequences in the input sequence and put them at proper places in the output sequence. Our empirical study on both synthetic data sets and real world data sets demonstrates the efficacy of CopyNet. For example, CopyNet can outperform regular RNN-based model with remarkable margins on text summarization tasks.
中文速览
序列到序列（Seq2Seq）模型在生成文本时常常需要直接"复制"输入中的片段——比如对话中重复对方说的人名或短语——但传统的编码器-解码器框架缺乏这种能力。为此，研究者提出了 CopyNet，在标准的注意力机制之外引入了一套"复制模式"：模型在每一步预测时，可以通过一个共享 softmax 在"从词表生成"和"从输入序列复制"之间自由切换，同时借助"选择性读取"让模型在连续复制多个词时能够自动追踪位置。整个框架完全可微，无需额外标注，可以端到端地用梯度下降训练。在文本摘要等真实任务上，CopyNet 显著优于普通 RNN 编码器-解码器，尤其在处理输入中的低频词和词表外词汇（OOV words）时优势明显。这项工作为后来大量依赖"复制-生成"混合机制的 NLP 模型奠定了基础，对对话系统、摘要生成、问答等需要忠实转述原文信息的场景具有重要的实践价值。
原文 arXiv:1603.06393;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1603.06393v3