arXiv:2512.14879 · 中英对照阅读
Entropy-Reservoir Bregman Projection: An Information-Geometric Unification of Model Collapse
中文速览
反复用模型自己生成的数据训练,可能让语言越来越重复、生成模型漏掉多样性,也让强化学习策略变得单一,但过去各种补救办法缺少统一解释。论文提出“熵库 Bregman 投影”(ERBP)框架,把这类训练看作模型不断贴近自己采样结果的循环,并在每轮掺入外部的高多样性数据或反馈,给系统补充“熵”——也就是变化和多样性。理论分析指出,缺少外部补充时,有限样本会让模型逐渐只保留越来越少的可能输出,最终走向崩塌;只要补充力度达到一定下限,就能维持多样性。作者还在语言模型自训练、强化学习和 GAN 实验中检验了这一解释,说明混入真实数据、鼓励探索等不同做法可以归结为同一原则。这个框架把零散的经验技巧变成了可分析、可调节的设计规则,提醒开发者持续监测并合理补足模型的多样性。
摘要
Self-referential learning—training a model on data it generated itself—promises boundless scalability but chronically suffers from model collapse: language models degenerate into repetitive text, GANs drop modes, and reinforcement-learning policies over-exploit. Although practitioners employ ad hoc fixes such as real-data mixing, entropy bonuses, knowledge distillation, or retrieval-augmented generation, a single principle that explains both the failure mode and the success of these fixes has remained elusive. We present Entropy-Reservoir Bregman Projection (ERBP), an information-geometric framework that unifies these phenomena. We model the closed loop as a stochastic Bregman projection sequence in distribution space. Without external coupling, finite-sample noise forces the system to project onto an ever-shrinking empirical support, causing exponential entropy decay and eventual collapse. Introducing an Entropy Reservoir—a high-entropy distribution mixed into each projection—injects a controllable entropy flux that provably stabilises the dynamics. Our theory yields (i) a necessary condition for collapse, (ii) a sufficient condition that guarantees a non-trivial entropy floor, and (iii) closed-form rates that depend only on sample size and the strong-convexity/Lipschitz constants of the Bregman generator. Experiments on large-language-model self-training, Soft Actor-Critic in reinforcement learning, and GAN optimisation validate our predictions and show that disparate stabilisation heuristics correspond to specific reservoir choices and coupling coefficients. ERBP thus transforms a collection of folk remedies into a single, quantitative design rule: monitor and budget your entropy flux.
术语表
- Self-referential learning
- 自指学习
- Model collapse
- 模型崩溃
- Entropy-Reservoir Bregman Projection (ERBP)
- 熵库—Bregman 投影(ERBP)
- Information geometry
- 信息几何
- Bregman projection
- Bregman 投影
- Bregman generator
- Bregman 生成函数
- Distribution space
- 分布空间
- Empirical support
- 经验支撑集
- Entropy decay
- 熵衰减
- Entropy Reservoir
- 熵库
- Entropy flux
- 熵通量
- Entropy floor
- 熵下界
- Strong convexity
- 强凸性
- Lipschitz constant
- Lipschitz 常数
- Generative degeneracy
- 生成退化
- Curse of recursion
- 递归诅咒
- Mode collapse
- 模式崩溃
- Policy collapse
- 策略崩溃
- Reinforcement learning (RL)
- 强化学习(RL)
- Generative adversarial network (GAN)
- 生成对抗网络(GAN)
- Large language model (LLM)
- 大语言模型(LLM)
- Soft Actor-Critic (SAC)
- Soft Actor-Critic(SAC)
- Maximum entropy reinforcement learning
- 最大熵强化学习
- Self-training
- 自训练
- Pseudo-labeling
- 伪标签
- Iterated learning
- 迭代学习
- Mirror Descent
- 镜像下降
- Natural Gradient Descent
- 自然梯度下降
- Retrieval-Augmented Generation (RAG)
- 检索增强生成(RAG)
- Reinforcement Learning from Human Feedback (RLHF)
- 基于人类反馈的强化学习(RLHF)