Aha.
正在载入中英对照阅读…

arXiv:2606.21090 · 中英对照阅读

Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training

Jianzhe Lin

中文速览

用强化学习反复训练代码模型时,分数可能先迅速上涨、再在同一轮训练里跌到接近零,论文要弄清怎样才能少掉分、让多轮训练的收益留下来。研究者用不同规模的 Qwen 模型做了多轮代码题实验,并比较三种办法:记录能力变化、决定是否沿用旧策略的 CARE,看到分数见顶就停训并保留最佳模型的早停(ES),以及改用另一种强化学习更新规则 GRPO。结果显示,没有一种办法在所有情况下都最好:CARE 对较脆弱的小模型帮助明显,早停能提高较大模型的表现,GRPO 也能让训练结果更稳,但它们都没能消除单轮训练中“先升后崩”的现象。研究说明,自我改进不只是选对奖励或算法,还要看模型在哪个训练阶段开始退步,并在合适的时机控制训练,才能避免把短期进步变成长期倒退。

摘要

Self-improvement can self-regress. In REINFORCE post-training for code, a model can improve rapidly on the very metric it optimizes and then collapse within the same training campaign. We document this failure mode in a controlled, multi-seed testbed: Qwen-2.5 at 3B and 7B, trained on competitive-programming tasks with binary CodeGrader reward over 10 sequential 20-step campaigns. Across campaigns, pass@1 exhibits a robust rise-then-collapse: it climbs to a peak in tens of gradient steps and then falls back, sometimes to near zero. This is not catastrophic forgetting across tasks; it is within-task policy over-optimization on a fixed distribution, and KL- and EWC-style parameter-level constraints do not prevent it.

术语表

self-improvement
自我改进
REINFORCE
REINFORCE
post-training
后训练
CodeGrader
CodeGrader
pass@1
pass@1
rise-then-collapse
先升后崩
within-task policy over-optimization
任务内策略过度优化
catastrophic forgetting
灾难性遗忘
KL constraint
KL约束
EWC
弹性权重巩固(EWC)
CARE
能力感知研究经验系统(CARE)
Capability-Effect Memory
能力效应记忆
Self-Improvement Transfer Gate
自我改进迁移门控
Regression-Aware Belief Revision
回归感知信念修正
early stopping (ES)
早停(ES)
GRPO
组相对策略优化(GRPO)
group-relative reward normalization
组相对奖励归一化
binary reward
二元奖励
verifiable rewards
可验证奖励
reinforcement learning with verifiable rewards (RLVR)
可验证奖励强化学习(RLVR)
policy entropy
策略熵
exploration
探索
rejection sampling
拒绝采样
solution diversity
解答多样性
capability vector
能力向量
capability regression through scalar optimization
标量优化导致的能力退化
bootstrap confidence interval
自助法置信区间
gradient step
梯度步