Measuring Faithfulness in Chain-of-Thought Reasoning
Tamera Lanham Anna Chen Ansh Radhakrishnan Benoit Steiner Carson Denison Danny Hernandez Dustin Li Esin Durmus Evan Hubinger Jackson Kernion Kamile Lukosiute Karina Nguyen Newton Cheng Nicholas Joseph Nicholas Schiefer Oliver Rausch Robin Larson Sam McCandlish Sandipan Kundu Saurav Kadavath Shannon Yang Thomas Henighan Timothy Maxwell Timothy Telleen-Lawton Tristan Hume Zac Hatfield-Dodds Jared Kaplan Jan Brauner Samuel R. Bowman Ethan Perez
Abstract
Large language models (LLMs) perform better when they produce step-by-step, “Chain-of-Thought” (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model’s actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT (e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT’s performance boost does not seem to come from CoT’s added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen.
中文速览
大型语言模型（LLM）在生成逐步推理（即"思维链"，Chain-of-Thought，CoT）后再给出答案时表现更好，但这些写出来的推理步骤是否真实反映了模型内部的决策过程，一直存疑。研究者通过截断推理链、故意插入错误、替换为无意义填充文本以及对推理进行同义改写等干预手段，系统测试了CoT的"忠实度"（faithfulness）。结果发现，模型对推理链的依赖程度因任务而异差别极大——在代数应用题等任务上推理链被大量使用，而在部分科学问答任务上几乎被完全忽略；此外，仅靠额外的推理token占位或特定措辞并不能解释CoT带来的性能提升。更令人意外的是，模型越大、能力越强，其推理链在多数任务上反而越不忠实，说明选择合适的模型规模和任务类型是获得可信CoT的关键。这一发现对依赖CoT做可解释性判断的高风险应用场景具有重要警示意义，也为未来设计更可靠的推理机制提供了方向。
原文 arXiv:2307.13702;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2307.13702v1