CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge
Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, Greg Durrett The University of Texas at Austin {yasumasa, mjqzhang, eunsol,
Abstract
Most benchmark datasets targeting commonsense reasoning focus on everyday scenarios: physical knowledge like knowing that you could fill a cup under a waterfall (Talmor et al., 2019), social knowledge like bumping into someone is awkward (Sap et al., 2019), and other generic situations. However, there is a rich space of commonsense inferences anchored to knowledge about specific entities: for example, deciding the truthfulness of a claim Harry Potter can teach classes on how to fly on a broomstick. Can models learn to combine entity knowledge with commonsense reasoning in this fashion? We introduce Creak, a testbed for commonsense reasoning about entity knowledge, bridging fact-checking about entities (Harry Potter is a wizard and is skilled at riding a broomstick) with commonsense inferences (if you’re good at a skill you can teach others how to do it). Our dataset consists of 13k human-authored English claims about entities that are either true or false, in addition to a small contrast set. Crowdworkers can easily come up with these statements and human performance on the dataset is high (high 90s); we argue that models should be able to blend entity knowledge and commonsense rea
中文速览
现有的常识推理数据集大多聚焦于日常生活场景，却忽视了一类需要将实体知识与常识推断相结合的判断——比如判断"哈利·波特能教别人骑扫帚飞行"这句话是否成立。为了填补这一空白，研究团队构建了 Creak 数据集：包含 1.3 万条由众包工作者撰写的英文真假声明，每条声明都围绕一个真实实体，要求模型同时调用实体事实知识与常识推理能力，而非依赖单一维基百科段落来核验。实验发现，在封闭书本设置（closed-book setting，即不借助外部检索）下，现有最强预训练语言模型与人类之间仍存在显著差距——在对比集上差距高达 27 个百分点，而在 Creak 上专门训练的模型也明显优于在其他事实核验数据集上训练的模型，表明该任务确实考察了不同于传统基准的推理能力。这项工作为评估语言模型"懂实体、会推断"的综合能力提供了一个干净、公平的测试平台，对推动更可靠的自然语言理解系统具有重要参考价值。
原文 arXiv:2109.01653;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.01653v1