Harvesting Paragraph-Level Question-Answer Pairs from Wikipedia
Xinya Du and Claire Cardie Department of Computer Science Cornell University Ithaca, NY, 14853, USA {xdu,
Abstract
We study the task of generating from Wikipedia articles question-answer pairs that cover content beyond a single sentence. We propose a neural network approach that incorporates coreference knowledge via a novel gating mechanism. Compared to models that only take into account sentence-level information Heilman and Smith (2010); Du et al. (2017); Zhou et al. (2017), we find that the linguistic knowledge introduced by the coreference representation aids question generation significantly, producing models that outperform the current state-of-the-art. We apply our system (composed of an answer span extraction system and the passage-level QG system) to the 10,000 top-ranking Wikipedia articles and create a corpus of over one million question-answer pairs. We also provide a qualitative analysis for this large-scale generated corpus from Wikipedia.
中文速览
如何让机器自动从维基百科文章里生成高质量的问答对，一直是个难题，尤其当一道题的答案需要跨越多个句子、靠代词指代（共指消解，coreference resolution）才能理解时，现有方法就力不从心了。研究者提出了一个叫 CorefNQG 的神经网络模型，核心创新是设计了一种门控机制（gating mechanism），把共指消解系统识别出的实体簇信息融入编码过程——简单说就是把"他""它"这类代词自动替换或标注回其所指的真实实体，再用分数加权控制这些信息对生成的影响程度。在 SQuAD 数据集上的评测表明，CorefNQG 明显优于只看单句的基线模型，在那些本就依赖跨句指代的题目子集上优势更为突出，整体刷新了当时的最优水平。研究团队进一步把这套系统跑遍维基百科排名前一万的文章，生成了超过一百万对问答，为阅读理解研究提供了一个大规模、低成本的新语料库。
原文 arXiv:1805.05942;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1805.05942v1