PIQA: Reasoning about Physical Commonsense in Natural Language
Yonatan Bisk1,2,3,4 Rowan Zellers1,4 Ronan Le Bras1 Jianfeng Gao2 Yejin Choi1,4 1Allen Institute for Artificial Intelligence 2Microsoft Research AI 3Carnegie Mellon University 4Paul G. Allen School for Computer Science and Engineering, University of Washington http://yonatanbisk.com/piqa
Abstract
To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today’s natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains – such as news articles and encyclopedia entries, where text is plentiful – in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical commonsense questions without experiencing the physical world?
中文速览
人工智能系统能否在不亲身接触物理世界的情况下，回答"用棉签还是牙签涂眼影"这类依赖常识的问题？为探究这一问题，研究者构建了名为 PIQA（Physical Interaction: Question Answering，物理交互问答）的基准数据集，包含逾16,000道二选一题目，题目来源于真实的生活技巧网站，并通过 AFLite 算法过滤掉了标注偏差。实验结果显示，人类作答准确率高达95%，而当时最强的预训练语言模型 RoBERTa 仅达到约77%，与人类水平相差近18个百分点。这一差距表明，仅靠从海量文本中学习语言规律，模型仍难以获得扎实的物理常识，该数据集因此为未来研究如何让 AI 真正理解现实世界提供了重要的评测基准和改进方向。
原文 arXiv:1911.11641;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.11641v1