Pointing the Unknown Words
Caglar Gulcehre Université de Montréal、Sungjin Ahn Université de Montréal、Ramesh Nallapati IBM T.J. Watson Research \ANDBowen Zhou IBM T.J. Watson Research、Yoshua Bengio Université de Montréal CIFAR Senior Fellow
Abstract
The problem of rare and unknown words is an important issue that can potentially effect the performance of many NLP systems, including both the traditional count-based and the deep learning models. We propose a novel way to deal with the rare and unseen words for the neural network models using attention. Our model uses two softmax layers in order to predict the next word in conditional language models: one predicts the location of a word in the source sentence, and the other predicts a word in the shortlist vocabulary. At each time-step, the decision of which softmax layer to use choose adaptively made by an MLP which is conditioned on the context. We motivate our work from a psychological evidence that humans naturally have a tendency to point towards objects in the context or the environment when the name of an object is not known. We observe improvements on two tasks, neural machine translation on the Europarl English to French parallel corpora and text summarization on the Gigaword dataset using our proposed model.
中文速览
神经网络做机器翻译、文本摘要等任务时，遇到生僻词或从未见过的词就会束手无策，通常只能用一个"未知词"符号（UNK）敷衍了事，严重损害输出质量。受人类"不知道名字就用手指一指"这一直觉行为的启发，论文提出了"指针Softmax（Pointer Softmax）"方法：模型同时维护两个输出层，一个从常用词表里选词，另一个像指针一样直接指向原文中的某个位置并把那个词复制过来，两者之间由一个小型神经网络动态决定每一步该用哪个。实验在英法机器翻译（Europarl语料库）和文本摘要（Gigaword数据集）上均取得了明显提升，证明该机制能有效处理生僻词和未登录词。这项工作的价值在于它既能与现有的大词表加速技巧无缝结合，又不像前人方法那样局限于固定的指针范围，具备更强的通用性。
原文 arXiv:1603.08148;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1603.08148v3