How Is ChatGPT’s Behavior Changing over Time?
Lingjiao Chen†, Matei Zaharia‡, James Zou† †Stanford University ‡UC Berkeley
Abstract
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers ( $84\%$ accuracy) but GPT-4 (June 2023) was poor on these same questions ( $51\%$ accuracy). This is partly explained by a drop in GPT-4’s amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5’s performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June th
中文速览
同样是"GPT-4"或"GPT-3.5"，短短几个月内它的表现可能已经大变样。研究者系统比较了这两款大语言模型（LLM）在2023年3月和6月两个版本上的表现，覆盖数学推理、敏感问题回答、代码生成、医学考试等七类任务，发现两个版本之间的行为差异相当惊人——例如GPT-4判断质数与合数的准确率从84%骤降至51%，而GPT-3.5在同一任务上却从50%跃升至76%。深入分析表明，GPT-4在6月版本中越来越不听从用户指令，包括不再按要求进行"链式思维（chain-of-thought）"推理，这是许多任务性能下滑的共同原因。这项研究揭示了一个容易被忽视的风险：把LLM服务嵌入实际系统时，"同一个"模型的悄然更新可能随时打乱下游流程，因此对线上LLM服务进行持续监测是保障可靠应用的必要前提。
原文 arXiv:2307.09009;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2307.09009v3