Can Large Language Models Infer Causation from Correlation?
Zhijing Jin1,2, ,‡‡\ddagger‡ Jiarui Liu3,††Zhiheng Lyu4 Spencer Poff5 Mrinmaya Sachan2 Rada Mihalcea6 Mona Diab3,‡‡\ddagger‡,††\dagger† Bernhard Schölkopf1,††\dagger† 1Max Planck Institute for Intelligent Systems, Tübingen, Germany 2ETH Zürich 3LTI, CMU 4University of Hong Kong 5Meta AI 6University of Michigan Equal contribution. ††\dagger†Equal supervision. ‡‡\ddagger‡Work originated as a Meta AI internship project involving Zhijing, Mona, and Spencer.
Abstract
Causal inference is one of the hallmarks of human intelligence. While the field of Causal NLP has attracted much interest in the recent years, existing causal inference datasets in NLP primarily rely on discovering causality from empirical knowledge (e.g., commonsense knowledge). In this work, we propose the first benchmark dataset to test the pure causal inference skills of large language models (LLMs). Specifically, we formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables. We curate a large-scale dataset of more than 200K samples, on which we evaluate seventeen existing LLMs. Through our experiments, we identify a key shortcoming of LLMs in terms of their causal inference skills, and show that these models achieve almost close to random performance on the task. This shortcoming is somewhat mitigated when we try to re-purpose LLMs for this skill via finetuning, but we find that these models still fail to generalize – they can only perform causal inference in in-distribution settings when variable names and textual expressions used in the queries are similar to those in the training set, but fai
中文速览
大语言模型（LLM）普遍被认为具备一定的因果推理能力，但现有评测几乎都依赖常识或经验知识，从未真正考察模型能否像人类统计学家那样，仅凭相关性信息（correlational statements）按照形式化规则推导出因果关系。研究者提出了一项全新任务 Corr2Cause——给定一组描述变量间相关性的陈述，判断某个因果关系假设是否成立——并基于经典因果发现（causal discovery）的 PC 算法自动生成了超过 20 万条标注样本。在这一基准上评测 17 个主流大语言模型后发现，所有模型的表现都接近随机猜测水平；即便对模型进行专项微调，性能有所提升，但一旦换用不同的变量名或表达方式（分布外场景），模型便立刻失效，说明它们学到的只是表面的文字模式而非真正的因果推理规则。这项工作揭示了当前大语言模型在纯形式化推理与泛化能力上的根本短板，为未来提升 LLM 的逻辑推理能力提供了一个具有挑战性的新基准。
原文 arXiv:2306.05836;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.05836v3