CRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models
Abstract
We introduce the CRASS (counterfactual reasoning assessment) data set and benchmark utilizing questionized counterfactual conditionals as a novel and powerful tool to evaluate large language models. We present the data set design and benchmark that supports scoring against a crowd-validated human baseline. We test six state-of-the-art models against our benchmark. Our results show that it poses a valid challenge for these models and opens up considerable room for their improvement. Keywords: common-sense reasoning, counterfactual conditionals, NLP, large language models
中文速览
大语言模型（LLM）在文本生成上越来越强，但在需要真正理解现实世界因果关系的常识推理任务上仍存在明显短板，现有基准测试又被攻克得越来越快，亟需新的评测工具。研究者为此构建了 CRASS 数据集与基准（counterfactual reasoning assessment，反事实推理评测），核心思路是把"如果当初发生了 X，会怎样"这类反事实条件句改写成问题，让模型从三个选项中选出最合理的结果，并以经过严格筛选的真人标注结果作为比较基线。实验对 GPT-3、T0pp 等六个顶尖模型进行测试，发现它们的表现比人类基线低 25% 以上，说明该任务对当前最强模型仍构成实质性挑战。这项工作不仅为评估 LLM 的因果与反事实推理能力提供了可靠的标准化工具，也已被纳入谷歌与 OpenAI 主导的 BIG-bench 评测体系，有助于持续推动语言模型在深层语言理解方向上的进步。
原文 arXiv:2112.11941;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.11941v3