Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanation
Cunxiang Wang1,4, Shuailong Liang2, Yue Zhang1, Xiaonan Li3 and Tian Gao4 1School of Engineering, Westlake University, China 2Singapore University of Technology and Design, Singapore 3School of Computer Science and Technology, Xidian University, China 4College of Computer Science and Technology, Zhejiang University, China
Abstract
Introducing common sense to natural language understanding systems has received increasing research attention. It remains a fundamental question on how to evaluate whether a system has a sense making capability. Existing benchmarks measures commonsense knowledge indirectly and without explanation. In this paper, we release a benchmark to directly test whether a system can differentiate natural language statements that make sense from those that do not make sense. In addition, a system is asked to identify the most crucial reason why a statement does not make sense. We evaluate models trained over large-scale language modeling tasks as well as human performance, showing that there are different challenges for system sense making.
中文速览
判断一句话"是否合乎常识"看似简单，却一直缺乏直接、可解释的评测基准，现有数据集大多通过阅读理解或事件预测等间接任务来衡量常识能力。为此，作者构建了一个名为 Sen-Making 的新基准，包含两个子任务：第一个任务要求模型从两句措辞相近的陈述中选出合乎常识的那句；第二个任务要求模型从三个候选理由中找出某句话"不合常识"的最关键原因，从而让评测既直接又可解释。实验结果显示，人类在两个子任务上的准确率分别高达 99.1% 和 97.3%，而在超过1亿句语料上训练的 ELMo 和 BERT 与人类水平仍存在显著差距，尤其在需要因果推理的解释子任务上表现更弱。这项工作的价值在于：它为自然语言理解系统提供了首个可直接量化常识推理能力、并同时要求给出决策依据的评测工具，有助于推动可解释常识推理的研究进展。
原文 arXiv:1906.00363;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.00363v2