Stochastic Answer Networks for Machine Reading Comprehension
Xiaodong Liu†, Yelong Shen†, Kevin Duh‡ and Jianfeng Gao† † Microsoft Research, Redmond, WA, USA ‡ Johns Hopkins University, Baltimore, MD, USA
Abstract
We propose a simple yet robust stochastic answer network (SAN) that simulates multi-step reasoning in machine reading comprehension. Compared to previous work such as ReasoNet which used reinforcement learning to determine the number of steps, the unique feature is the use of a kind of stochastic prediction dropout on the answer module (final layer) of the neural network during the training. We show that this simple trick improves robustness and achieves results competitive to the state-of-the-art on the Stanford Question Answering Dataset (SQuAD), the Adversarial SQuAD, and the Microsoft MAchine Reading COmprehension Dataset (MS MARCO).
中文速览
机器阅读理解（machine reading comprehension）任务要求模型读懂一段文字并回答相关问题，难点在于有些问题需要跨句子的多步推理才能找到正确答案。研究者提出了一种随机答案网络（Stochastic Answer Network, SAN），其核心创新在于训练时对最终预测层随机丢弃（dropout）某些推理步骤的输出，解码时则对所有步骤的预测取平均，而不是只依赖最后一步或某个动态选定的步骤。这一看似简单的"随机预测丢弃"技巧迫使模型在每一步都学会给出可靠答案，避免对某单一步骤过度依赖，相当于对模型自身的多步推理过程做了一种集成（ensemble）。实验结果表明，SAN 在 SQuAD、对抗性 SQuAD 以及 MS MARCO 三个权威数据集上均达到当时领先水平，且明显优于固定步数记忆网络和使用强化学习动态决定步数的 ReasoNet，证明该方法既有效又易于实现。
原文 arXiv:1712.03556;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1712.03556v2