CoQA: A Conversational Question Answering Challenge
Siva Reddy∗ Danqi Chen∗ Christopher D. Manning Computer Science Department Stanford University
Abstract
Humans gather information through conversations involving a series of interconnected questions and answers. For machines to assist in information gathering, it is therefore essential to enable them to answer conversational questions. We introduce CoQA, a novel dataset for building Conversational Question Answering systems.111CoQA is pronounced as coca. Our dataset contains 127k questions with answers, obtained from 8k conversations about text passages from seven diverse domains. The questions are conversational, and the answers are free-form text with their corresponding evidence highlighted in the passage. We analyze CoQA in depth and show that conversational questions have challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning. We evaluate strong dialogue and reading comprehension models on CoQA. The best system obtains an F1 score of 65.4%, which is 23.4 points behind human performance (88.8%), indicating there is ample room for improvement. We present CoQA as a challenge to the community at https://stanfordnlp.github.io/coqa.
中文速览
机器要真正帮人"聊着找答案"，就必须理解对话中层层递进的问题——后一个问题往往只有一两个字，离开上下文根本无从作答。为此，研究者构建了 CoQA 这个对话式问答数据集（Conversational Question Answering），从儿童故事、文学、新闻、维基百科等七个领域收集了 8000 段对话、共 12.7 万轮问答，每个答案都是自由文本，同时标注了段落中对应的证据片段。分析发现，CoQA 中近一半的问题涉及指代消解（coreference），超过四分之一需要依赖常识或语用推理，这些都是现有阅读理解数据集几乎不涉及的难点。测试结果表明，最强的基线系统 F1 值仅为 65.4%，而人类可以达到 88.8%，相差 23.4 个百分点，说明让机器真正融入自然对话还有很长的路要走，CoQA 也因此成为推动该领域发展的重要挑战性基准。
原文 arXiv:1808.07042;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1808.07042v2