arXiv:2606.21090 · 中英对照阅读
Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training
中文速览
用强化学习反复训练代码模型时,分数可能先迅速上涨、再在同一轮训练里跌到接近零,论文要弄清怎样才能少掉分、让多轮训练的收益留下来。研究者用不同规模的 Qwen 模型做了多轮代码题实验,并比较三种办法:记录能力变化、决定是否沿用旧策略的 CARE,看到分数见顶就停训并保留最佳模型的早停(ES),以及改用另一种强化学习更新规则 GRPO。结果显示,没有一种办法在所有情况下都最好:CARE 对较脆弱的小模型帮助明显,早停能提高较大模型的表现,GRPO 也能让训练结果更稳,但它们都没能消除单轮训练中“先升后崩”的现象。研究说明,自我改进不只是选对奖励或算法,还要看模型在哪个训练阶段开始退步,并在合适的时机控制训练,才能避免把短期进步变成长期倒退。
摘要
Self-improvement can self-regress. In REINFORCE post-training for code, a model can improve rapidly on the very metric it optimizes and then collapse within the same training campaign. We document this failure mode in a controlled, multi-seed testbed: Qwen-2.5 at 3B and 7B, trained on competitive-programming tasks with binary CodeGrader reward over 10 sequential 20-step campaigns. Across campaigns, pass@1 exhibits a robust rise-then-collapse: it climbs to a peak in tens of gradient steps and then falls back, sometimes to near zero. This is not catastrophic forgetting across tasks; it is within-task policy over-optimization on a fixed distribution, and KL- and EWC-style parameter-level constraints do not prevent it.
术语表
- self-improvement
- 自我改进
- REINFORCE
- REINFORCE
- post-training
- 后训练
- CodeGrader
- CodeGrader
- pass@1
- pass@1
- rise-then-collapse
- 先升后崩
- within-task policy over-optimization
- 任务内策略过度优化
- catastrophic forgetting
- 灾难性遗忘
- KL constraint
- KL约束
- EWC
- 弹性权重巩固(EWC)
- CARE
- 能力感知研究经验系统(CARE)
- Capability-Effect Memory
- 能力效应记忆
- Self-Improvement Transfer Gate
- 自我改进迁移门控
- Regression-Aware Belief Revision
- 回归感知信念修正
- early stopping (ES)
- 早停(ES)
- GRPO
- 组相对策略优化(GRPO)
- group-relative reward normalization
- 组相对奖励归一化
- binary reward
- 二元奖励
- verifiable rewards
- 可验证奖励
- reinforcement learning with verifiable rewards (RLVR)
- 可验证奖励强化学习(RLVR)
- policy entropy
- 策略熵
- exploration
- 探索
- rejection sampling
- 拒绝采样
- solution diversity
- 解答多样性
- capability vector
- 能力向量
- capability regression through scalar optimization
- 标量优化导致的能力退化
- bootstrap confidence interval
- 自助法置信区间
- gradient step
- 梯度步