Attention-over-Attention Neural Networks for Reading Comprehension
Yiming Cui†, Zhipeng Chen†, Si Wei†, Shijin Wang†, Ting Liu‡ Guoping Hu† †Joint Laboratory of HIT and iFLYTEK, iFLYTEK Research, Beijing, China ‡Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China
Abstract
Cloze-style reading comprehension is a representative problem in mining relationship between document and query. In this paper, we present a simple but novel model called attention-over-attention reader for better solving cloze-style reading comprehension task. The proposed model aims to place another attention mechanism over the document-level attention and induces “attended attention” for final answer predictions. One advantage of our model is that it is simpler than related works while giving excellent performance. In addition to the primary model, we also propose an N-best re-ranking strategy to double check the validity of the candidates and further improve the performance. Experimental results show that the proposed methods significantly outperform various state-of-the-art systems by a large margin in public datasets, such as CNN and Children’s Book Test.
中文速览
完形填空式阅读理解（cloze-style reading comprehension）要求机器读懂文章并从中找到填入问句空白处的正确词汇，难点在于如何同时利用文章和问句的交互信息。研究者提出了一种名为"注意力之上的注意力"（Attention-over-Attention, AoA）的新型神经网络模型：先计算每个问句词对文章各位置的注意力分布，再反过来让文章每个词对问句词打分，两者加权融合，使模型既能从问句看文章，也能从文章看问句，自动决定问句中哪些词更值得参考。此外，他们还设计了一种N-best重排序策略，将候选答案回填进问句后，用语言模型从流畅度等多个角度再次打分，进一步筛选最优答案。实验表明，该方法在CNN和Children's Book Test两个公开数据集上大幅超越当时所有已知系统，而模型结构本身比同类方法更简洁，证明了"叠加注意力"这一思路的有效性与实用价值。
原文 arXiv:1607.04423;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1607.04423v4