Aha.
正在载入中英对照阅读…

arXiv:2512.14879 · 中英对照阅读

Entropy-Reservoir Bregman Projection: An Information-Geometric Unification of Model Collapse

Jingwei Chen

中文速览

反复用模型自己生成的数据训练,可能让语言越来越重复、生成模型漏掉多样性,也让强化学习策略变得单一,但过去各种补救办法缺少统一解释。论文提出“熵库 Bregman 投影”(ERBP)框架,把这类训练看作模型不断贴近自己采样结果的循环,并在每轮掺入外部的高多样性数据或反馈,给系统补充“熵”——也就是变化和多样性。理论分析指出,缺少外部补充时,有限样本会让模型逐渐只保留越来越少的可能输出,最终走向崩塌;只要补充力度达到一定下限,就能维持多样性。作者还在语言模型自训练、强化学习和 GAN 实验中检验了这一解释,说明混入真实数据、鼓励探索等不同做法可以归结为同一原则。这个框架把零散的经验技巧变成了可分析、可调节的设计规则,提醒开发者持续监测并合理补足模型的多样性。

摘要

Self-referential learning—training a model on data it generated itself—promises boundless scalability but chronically suffers from model collapse: language models degenerate into repetitive text, GANs drop modes, and reinforcement-learning policies over-exploit. Although practitioners employ ad hoc fixes such as real-data mixing, entropy bonuses, knowledge distillation, or retrieval-augmented generation, a single principle that explains both the failure mode and the success of these fixes has remained elusive. We present Entropy-Reservoir Bregman Projection (ERBP), an information-geometric framework that unifies these phenomena. We model the closed loop as a stochastic Bregman projection sequence in distribution space. Without external coupling, finite-sample noise forces the system to project onto an ever-shrinking empirical support, causing exponential entropy decay and eventual collapse. Introducing an Entropy Reservoir—a high-entropy distribution mixed into each projection—injects a controllable entropy flux that provably stabilises the dynamics. Our theory yields (i) a necessary condition for collapse, (ii) a sufficient condition that guarantees a non-trivial entropy floor, and (iii) closed-form rates that depend only on sample size and the strong-convexity/Lipschitz constants of the Bregman generator. Experiments on large-language-model self-training, Soft Actor-Critic in reinforcement learning, and GAN optimisation validate our predictions and show that disparate stabilisation heuristics correspond to specific reservoir choices and coupling coefficients. ERBP thus transforms a collection of folk remedies into a single, quantitative design rule: monitor and budget your entropy flux.

术语表

Self-referential learning
自指学习
Model collapse
模型崩溃
Entropy-Reservoir Bregman Projection (ERBP)
熵库—Bregman 投影(ERBP)
Information geometry
信息几何
Bregman projection
Bregman 投影
Bregman generator
Bregman 生成函数
Distribution space
分布空间
Empirical support
经验支撑集
Entropy decay
熵衰减
Entropy Reservoir
熵库
Entropy flux
熵通量
Entropy floor
熵下界
Strong convexity
强凸性
Lipschitz constant
Lipschitz 常数
Generative degeneracy
生成退化
Curse of recursion
递归诅咒
Mode collapse
模式崩溃
Policy collapse
策略崩溃
Reinforcement learning (RL)
强化学习(RL)
Generative adversarial network (GAN)
生成对抗网络(GAN)
Large language model (LLM)
大语言模型(LLM)
Soft Actor-Critic (SAC)
Soft Actor-Critic(SAC)
Maximum entropy reinforcement learning
最大熵强化学习
Self-training
自训练
Pseudo-labeling
伪标签
Iterated learning
迭代学习
Mirror Descent
镜像下降
Natural Gradient Descent
自然梯度下降
Retrieval-Augmented Generation (RAG)
检索增强生成(RAG)
Reinforcement Learning from Human Feedback (RLHF)
基于人类反馈的强化学习(RLHF)