QED: A Framework and Dataset for Explanations in Question Answering
Matthew Lamm1 , Jennimaria Palomaki2, Chris Alberti2, Daniel Andor2, Eunsol Choi3 , Livio Baldini Soares2, Michael Collins2 1Department of Linguistics, Stanford University 2Google Research 3Department of Computer Science, The University of Texas at Austin {jpalomaki, chrisalberti, danielandor, liviobs, Work done during internship at Google. Work done at Google.
Abstract
A question answering system that in addition to providing an answer provides an explanation of the reasoning that leads to that answer has potential advantages in terms of debuggability, extensibility and trust. To this end, we propose QED, a linguistically informed, extensible framework for explanations in question answering. A QED explanation specifies the relationship between a question and answer according to formal semantic notions such as referential equality, sentencehood, and entailment. We describe and publicly release an expert-annotated dataset of QED explanations built upon a subset of the Google Natural Questions dataset, and report baseline models on two tasks – post-hoc explanation generation given an answer, and joint question answering and explanation generation. In the joint setting, a promising result suggests that training on a relatively small amount of QED data can improve question answering. In addition to describing the formal, language-theoretic motivations for the QED approach, we describe a large user study showing that the presence of QED explanations significantly improves the ability of untrained raters to spot errors made by a strong neural QA baselin
中文速览
问答系统（QA system）给出答案却不解释理由，让人难以判断对错——这篇论文就是要解决这个"只给答案、不给推理过程"的问题。作者提出了一套名为QED的解释框架，把问题与答案之间的关系拆解为三个语言学上有明确定义的步骤：找到蕴含答案的单句、标注问题与该句中指称同一事物的名词短语、以及确认句子谓词在抽象掉指称后蕴含问题谓词。基于此框架，作者在Google Natural Questions数据集上构建并公开发布了一个专家标注的QED语料库（约7600条训练样本），并训练了基线模型分别完成"事后生成解释"和"同步回答与解释生成"两项任务，结果显示即便只用少量QED数据联合训练也能提升问答准确率。更重要的是，大规模用户研究表明，有QED解释的辅助能显著帮助普通用户识别强神经网络QA模型犯下的错误，这说明可解释问答不仅在技术上可行，在实际可信赖性和人机协作层面也具有切实价值。
原文 arXiv:2009.06354;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2009.06354v1