MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
Boyuan Pan△∗subscriptsuperscriptabsent△{}^{*}_{\bigtriangleup}, Hao Li△∗subscriptsuperscriptabsent△{}^{*}_{\bigtriangleup}, Zhou Zhao†, Bin Cao‡, Deng Cai∗, Xiaofei He∗ ∗State Key Lab of CAD、\、CG, College of Computer Science, Zhejiang University, Hangzhou, China †College of Computer Science, Zhejiang University, Hangzhou, China ‡Eigen Technologies, Hangzhou, China {{\{panby, haolics, {{\{dengcai,
Abstract
Machine comprehension(MC) style question answering is a representative problem in natural language processing. Previous methods rarely spend time on the improvement of encoding layer, especially the embedding of syntactic information and name entity of the words, which are very crucial to the quality of encoding. Moreover, existing attention methods represent each query word as a vector or use a single vector to represent the whole query sentence, neither of them can handle the proper weight of the key words in query sentence. In this paper, we introduce a novel neural network architecture called Multi-layer Embedding with Memory Network(MEMEN) for machine reading task. In the encoding layer, we employ classic skip-gram model to the syntactic and semantic information of the words to train a new kind of embedding layer. We also propose a memory network of full-orientation matching of the query and passage to catch more pivotal information. Experiments show that our model has competitive results both from the perspectives of precision and efficiency in Stanford Question Answering Dataset(SQuAD) among all published results and achieves the state-of-the-art results on TriviaQA dataset.
中文速览
机器阅读理解任务要求模型读懂一段文字后回答相关问题，此前主流方法大多只关注注意力机制的改进，忽视了词语编码层的质量，尤其是句法标签（词性，POS）和命名实体识别（NER）信息的利用，同时现有的一维或二维注意力各有缺陷，难以兼顾关键词的权重。针对这些不足，研究者提出了名为 MEMEN（Multi-layer Embedding with Memory Network）的端到端神经网络模型：在编码层用 skip-gram 模型为 POS 和 NER 标签训练专属词向量，丰富词语的句法与语义表示；在匹配层设计"全方位匹配"记忆网络，将一维整体查询匹配、二维逐词相似度匹配和基于上下文的匹配融合为多跳记忆结构，并加入门控机制加速收敛。实验结果显示，该模型在 TriviaQA 数据集上达到当时最优水平，在 SQuAD 数据集上取得 75.37% 精确匹配和 82.66% F1 的有竞争力成绩，且省去了计算代价高昂的自匹配机制，训练效率更高。这项工作表明，在编码层融入更丰富的语言学特征、并通过多角度匹配深化模型对问题与段落关系的理解，是提升阅读理解性能的有效途径。
原文 arXiv:1707.09098;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.09098v1