When do you need Chain-of-Thought Prompting for ChatGPT?
Jiuhai Chen* University of Maryland \AndLichang Chen* University of Maryland \AndHeng Huang University of Maryland \AndTianyi Zhou University of Maryland
Abstract
Chain-of-Thought (CoT) prompting can effectively elicit complex multi-step reasoning from Large Language Models (LLMs). For example, by simply adding CoT instruction “Let’s think step-by-step” to each input query of MultiArith dataset, GPT-3’s accuracy can be improved from 17.7% to 78.7%. However, it is not clear whether CoT is still effective on more recent instruction finetuned (IFT) LLMs such as ChatGPT. Surprisingly, on ChatGPT, CoT is no longer effective for certain tasks such as arithmetic reasoning while still keeping effective on other reasoning tasks. Moreover, on the former tasks, ChatGPT usually achieves the best performance and spontaneously generates CoT even without being instructed to do so. Hence, it is plausible that ChatGPT has already been trained on these tasks with CoT and thus memorized the instruction so it implicitly follows such an instruction when applied to the same queries, even without CoT. Our analysis reflects a potential risk of overfitting/bias toward instructions introduced in IFT, which becomes more common in training LLMs. In addition, it indicates possible leakage of the pretraining recipe, e.g., one can verify whether a dataset and instruction
中文速览
链式思维提示（Chain-of-Thought, CoT）曾被证明能大幅提升GPT-3等大语言模型的推理能力，但当模型升级为经过指令微调的ChatGPT后，这一技巧是否仍然奏效却成了未解之谜。研究者在算术、常识、符号等多类推理任务上系统对比了"无指令""有CoT指令"两种零样本策略，结果令人惊讶：ChatGPT在算术推理任务上即使不加任何指令，也会自发生成逐步推理过程，而强行加上CoT提示反而会拉低准确率；但在其他推理任务上，ChatGPT与GPT-3表现一致，依然从CoT中受益。对此，作者提出一个合理解释：ChatGPT在指令微调阶段已经接触过这些算术任务与CoT指令的配对数据，并将其"记住"，导致遇到相同类型问题时会隐式套用该指令，显式提示反而造成干扰。这一发现不仅揭示了指令微调可能带来的过拟合风险，还表明仅通过黑盒查询就能推断某个数据集和指令是否曾被用于模型训练，为大语言模型的训练数据审计和安全研究提供了新思路。
原文 arXiv:2304.03262;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.03262v2