Can Large Language Models Build Causal Graphs?
Stephanie Long Dept. of Family Medicine, McGill University、Tibor Schuster Dept. of Family Medicine, McGill University、Alexandre Piché Mila, Université de Montréal ServiceNow Research
Abstract
Building causal graphs can be a laborious process. To ensure all relevant causal pathways have been captured, researchers often have to discuss with clinicians and experts while also reviewing extensive relevant medical literature. By encoding common and medical knowledge, large language models (LLMs) represent an opportunity to ease this process by automatically scoring edges (i.e., connections between two variables) in potential graphs. LLMs however have been shown to be brittle to the choice of probing words, context, and prompts that the user employs. In this work, we evaluate if LLMs can be a useful tool in complementing causal graph development.
中文速览
构建因果有向无环图（DAG）是医学研究中的关键步骤，但需要大量专家讨论和文献梳理，耗时费力。研究者尝试用大型语言模型GPT-3来自动判断两个医学变量之间是否存在因果连接，从而辅助这一过程。实验基于四张涵盖糖尿病、肥胖、癌症和酒精等主题的标准医学DAG，系统测试了不同提示语（如"据医生表示"）、不同连接动词（如"导致"或"增加风险"）以及不同变量描述细节对GPT-3判断准确率的影响。结果显示GPT-3在多数设置下的准确率明显高于随机猜测，但表现对措辞高度敏感——没有哪种提示或动词在所有场景下都一致有效，"Big Pharma"提示甚至会降低准确率。这说明LLM目前尚不能独立替代专家构建DAG，但作为辅助工具提炼既有医学知识、缩短建图流程具有实际潜力，前提是必须有领域专家进行验证把关。
原文 arXiv:2303.05279;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.05279v2