arXiv:2605.14831 · 中英对照阅读
有趣性作为未来压缩进展的归纳式启发法
Interestingness as an Inductive Heuristic for Future Compression Progress
中文速览
要让能自我改进的系统持续进步,关键不只是从数据里学习,还得提前判断哪些任务和材料值得投入计算。作者把“有趣”定义成对未来压缩进步的预测,并用不同的概率先验分析复杂度随运行时间变化的轨迹。理论和三种通用计算系统上的实验都显示,过去的压缩突破确实能提示未来进展;但距离上次突破越久,近期再有突破的预期会按指数下降,而算法先验比长度先验对未来更乐观,同样的历史轨迹下预期发现量可高出约一倍。这个结果为自主生成学习任务提供了可检验的筛选思路,说明系统或许能依据自身的进步历史,优先探索仍可能带来新发现的方向。
摘要
迈向递归自我改进系统的瓶颈之一,是有趣性这一挑战:前瞻性地识别哪些任务或数据蕴含未来进展潜力的能力。我们将有趣性形式化为一种用于未来压缩进展的归纳式启发法,并借助 Kolmogorov复杂度(Kolmogorov Complexity)和算法统计学(Algorithmic Statistics)中的工具,研究其可预测性。通过分析长度先验(Length Prior)、算法先验(Algorithmic Prior)和速度先验(Speed Prior)下的复杂度—运行时间特征(complexity-runtime profile),我们证明,有趣性的归纳属性——即过去的进展能够预示未来发现的能力——在理论上可行,并得到经验支持。我们证明,未来预期进展取决于距上次观察到突破的时间远近,且呈指数关系。此外,我们表明,对于相同的观测特征,算法先验比长度先验乐观得多,预期发现量增加至原来的平方。这些发现通过三种多样化的通用计算范式上的实验得到了证实。
One of the bottlenecks on the way towards recursively self-improving systems is the challenge of interestingness: the ability to prospectively identify which tasks or data hold the potential for future progress. We formalize interestingness as an inductive heuristic for future compression progress and investigate its predictability using tools from Kolmogorov Complexity and Algorithmic Statistics. By analyzing complexity-runtime profiles under Length, Algorithmic, and Speed priors, we demonstrate that the inductive property of interestingness—the capacity for past progress to signal future discovery—is theoretically viable and empirically supported. We prove that expected future progress depends exponentially on the recency of the last observed breakthrough. Furthermore, we show that the Algorithmic Prior is significantly more optimistic than the Length Prior, yielding a quadratic increase in expected discovery for the same observed profile. These findings are experimentally confirmed across three diverse universal computational paradigms.
术语表
- interestingness
- 有趣性
- recursive self-improvement
- 递归自我改进
- open-ended intelligence
- 开放式智能
- inductive heuristic
- 归纳式启发法
- compression progress
- 压缩进展
- Kolmogorov Complexity
- Kolmogorov复杂度
- Algorithmic Statistics
- 算法统计学
- complexity-runtime profile
- 复杂度—运行时间特征
- Length Prior
- 长度先验
- Algorithmic Prior
- 算法先验
- Speed Prior
- 速度先验
- stagnation length
- 停滞时长
- compression breakthrough
- 压缩突破
- Large Language Model (LLM)
- 大语言模型(LLM)
- Reinforcement Learning (RL)
- 强化学习(RL)
- model collapse
- 模型崩溃
- intrinsic motivation
- 内在动机
- count-based exploration
- 基于计数的探索
- space coverage
- 空间覆盖
- maximum entropy
- 最大熵
- prediction error
- 预测误差
- epistemic uncertainty
- 认知不确定性
- aleatoric uncertainty
- 偶然不确定性
- Noisy TV problem
- 噪声电视问题
- information gain
- 信息增益
- Bayesian surprise
- 贝叶斯惊奇度
- Kullback-Leibler divergence
- Kullback–Leibler散度
- learning progress
- 学习进展
- competence progress
- 能力进展
- negative log-likelihood (NLL)
- 负对数似然(NLL)