Ruminating Reader: Reasoning with Gated Multi-Hop Attention
Yichen Gong Samuel R. Bowman New York University New York, NY
Abstract
To answer the question in machine comprehension (MC) task, the models need to establish the interaction between the question and the context. To tackle the problem that the single-pass model cannot reflect on and correct its answer, we present Ruminating Reader. Ruminating Reader adds a second pass of attention and a novel information fusion component to the Bi-Directional Attention Flow model (BiDAF). We propose novel layer structures that construct an query-aware context vector representation and fuse encoding representation with intermediate representation on top of BiDAF model. We show that a multi-hop attention mechanism can be applied to a bi-directional attention structure. In experiments on SQuAD, we find that the Reader outperforms the BiDAF baseline by a substantial margin, and matches or surpasses the performance of all other published systems.
中文速览
机器阅读理解（Machine Comprehension）模型在回答问题时，往往只对文章和问题"过一遍"，一旦第一次判断出错就难以纠正。为此，研究者提出了"Ruminating Reader"，在经典的双向注意力流模型（BiDAF）基础上增加了第二轮注意力机制，并设计了两个新的"反刍层"（ruminate layers），通过门控机制将第一轮和第二轮读取所得的表示融合起来，让模型能像人类多次阅读一样，对问题和文章进行更深层的交互与修正；此外还引入了一个答案-问题相似度惩罚损失，抑制模型把与问题高度重叠的片段误选为答案。在斯坦福问答数据集（SQuAD）上，该模型的F1分数达到79.5、精确匹配（EM）达到70.6，相比BiDAF基线分别提升2.2和2.9个百分点，并在当时超越或持平所有已发表系统。这项工作表明，在双向注意力结构上叠加多跳（multi-hop）注意力是切实可行的，为后续多轮推理阅读理解模型的发展提供了重要参考。
原文 arXiv:1704.07415;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1704.07415v1