Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training

摘要

Self-improvement can self-regress. In REINFORCE post-training for code, a model can improve rapidly on the very metric it optimizes and then collapse within the same training campaign. We document this failure mode in a controlled, multi-seed testbed: Qwen-2.5 at 3B and 7B, trained on competitive-programming tasks with binary CodeGrader reward over 10 sequential 20-step campaigns. Across campaigns, pass@1 exhibits a robust rise-then-collapse: it climbs to a peak in tens of gradient steps and then falls back, sometimes to near zero. This is not catastrophic forgetting across tasks; it is within-task policy over-optimization on a fixed distribution, and KL- and EWC-style parameter-level constraints do not prevent it.