Challenges in Information-Seeking QA: Unanswerable Questions and Paragraph Retrieval
Akari Asai University of Washington、Eunsol Choi111The work started while the author is visiting Google AI. The University of Texas at Austin
Abstract
Recent pretrained language models “solved” many reading comprehension benchmarks, where questions are written with the access to the evidence document. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging. We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA. Our controlled experiments suggest two headrooms – paragraph selection and answerability prediction, i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in information-seeking question an
中文速览
预训练语言模型在阅读理解基准上表现亮眼，却在"信息寻求型问答"（information-seeking QA）上仍远落后于人类——这类任务中，用户是真的不知道答案才来提问，问题与文档是独立产生的。研究者以 Natural Questions 和 TyDi QA 两个多语言数据集为分析对象，通过受控实验发现：模型的主要瓶颈集中在两个子问题上——从数十个段落中找到正确段落，以及判断文档是否包含答案（即"可回答性预测"）。一旦给模型提供正确段落和正确的答案类型，现有模型就能轻松超越单个人类标注者；然而在不借助任何辅助信息的情况下，可回答性预测本身依然困难。为此，研究者跨六种语言手工标注了 800 条不可回答样本，将其归入六个成因类别并逐类分析，揭示了现有数据集构建方式与任务定义中的固有缺陷，并据此为未来的数据集设计和模型开发指出了具体的改进方向。
原文 arXiv:2010.11915;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.11915v2