BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Christopher Clark∗1, Kenton Lee†, Ming-Wei Chang†, Tom Kwiatkowski† \ANDMichael Collins †2, Kristina Toutanova† ∗Paul G. Allen School of CSE, University of Washington †Google AI Language {kentonl, mingweichang, tomkwiat, mjcollins,
Abstract
In this paper we study yes/no questions that are naturally occurring — meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.
中文速览
真实用户在搜索引擎里自发提出的是/否问题（yes/no questions）听起来简单，实则暗藏复杂推理，现有模型对此应对能力并不清楚。研究团队从谷歌真实搜索记录中收集了1.6万条这类自然发生的是/否问题，配上维基百科段落，构建了名为BoolQ的阅读理解数据集，并通过人工分析证明这些问题往往需要超越字面的推理能力，而非简单的事实匹配。他们系统测试了多种迁移学习方案，发现先在自然语言推断数据集MultiNLI上微调、再在BoolQ上继续训练的BERT模型效果最佳，达到80.4%准确率，但仍显著低于人类标注者90%的水平。这一结果表明，即便是当时最强的预训练模型BERT，面对来自真实世界的是/否问题依然存在明显不足，BoolQ因此为评估模型深层语言理解能力提供了一个高难度却贴近实用场景的基准。
原文 arXiv:1905.10044;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.10044v1