Aha.
正在载入中英对照阅读…

arXiv:2601.05280 · 中英对照阅读

大型语言模型(LLM)是 Shannon 有损压缩器,而非 Solomonoff 归纳估计器:没有符号模型合成,递归自我改进与奇点尚远

Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesis

Hector Zenil、Abicumaran Uthamacumaran、Luan Ozelim

中文速览

只靠“猜下一个词”训练的大语言模型,是否能自然学成索罗门诺夫归纳、并靠反复训练实现自我进步?作者的结论是否定的:交叉熵目标只要求模型贴合训练数据的概率分布,不会自动让它按程序复杂度给各种解释分配权重,也没有标准来判断下一代模型是否真的更好。作者一方面分析这种训练目标和递归自训练的理论边界,另一方面测试了 26 个不同规模的模型,发现它们预测程序生成的数字序列时,表现都没达到一种简单通用预测方法的基准,模型变大或经过后训练也没显示出朝通用归纳靠近。论文因此强调,可靠的自我改进不能只靠生成更多数据或复制自身,还需要程序搜索、模型合成,以及实验、验证等独立标准来筛选改动;这为判断 AI 是否真能持续改进,而不只是变得更像自己,提供了更清晰的尺度。

摘要

我们将算法信息论(AIT)、机器学习(ML)与通用人工智能(AGI)中的两个问题联系起来:大型语言模型(LLM)是否在估计 Solomonoff 归纳,以及它们能否自我改进并迈向人工智能奇点。我们从理论、方法和实证层面给出否定答案,同时说明如何绕过这些限制。交叉熵、负对数似然及相关的下一词元预测目标本身无法实现 Solomonoff 归纳:它们拟合的是给定的条件分布,而非程序加权通用混合分布。更多计算可以改善固定目标下的拟合,但若没有外部超参数或架构调优,就无法改变其归纳原则;仅凭这些目标,也无法实现 Solomonoff-Levin 最优预测。对于有限学习器和观察者,理论边界的决定性较弱,不同方法也会出现分歧。资源受限估计器是有限的机制搜索工具,其分歧并不违反算法信息守恒。我们考察了五个预训练系列中二十六个已发布的语言模型检查点,其参数量为 0.8 至 350 亿,每个权重携带 1.9 至 8.5 比特。我们将它们作为下一位数字预测器,在属于一类明确定义的 177,591 个程序的上下文内数据流上进行测试。结果显示,它们会为该类程序永远不会产生的后续序列赋予概率质量,而且没有任何模型能够将与权重为 0.10 的 Krichevsky-Trofimov 编码混合的分布所落后的比特数控制在 3.32 比特以内:在二十四个符号时,表现最佳者落后 4.5 比特,中位数落后 36.5 比特,最差者落后 128 比特。在 256 个符号时,最接近该界限的三个检查点趋于落后 7 至 13 比特,这与以较小权重纳入该编码的混合分布相符;其他模型则持续落后。最接近 Solomonoff 概率分配的读法所追踪的是模型赋予字母表之外的概率质量,因此,现有结果并未表明,无论是扩大规模还是后训练,都能使模型趋近程序加权混合分布。前沿开发者正在采用神经符号方法,包括 Fable 和 Astra,并通过神经符号计算纳入模型合成。它们已不再是纯统计型大型语言模型(LLM),因此是探索更高阶归纳与模型合成的更好候选者,尽管仍存在局限。我们的结果与近期文献相矛盾;那些文献依据与纯统计压缩算法相关的论证,认为大型语言模型(LLM)是 Solomonoff 归纳估计器。

We connect two questions in Algorithmic Information Theory (AIT), Machine Learning (ML) and Artificial General Intelligence (AGI): whether LLMs estimate Solomonoff induction, and whether they can self-improve towards an AI Singularity. We provide theoretical, methodological and empirical answers in the negative but show how limits can be circumvented. Cross-entropy, negative log-likelihood and related next-token objectives cannot alone implement Solomonoff induction: they fit supplied conditionals rather than a program-weighted universal mixture. More computation can improve fit within a fixed objective but cannot change its inductive principle without external hyperparameter or architectural tuning; they alone do not deliver Solomonoff-Levin optimal prediction. For finite learners and observers, theoretical boundaries become less decisive and approaches diverge. Resource-bounded estimators are finite mechanism-search tools whose divergence does not violate algorithmic information conservation. Twenty-six served language-model checkpoints across five pre-training families, 0.8-35 billion parameters and 1.9-8.5 bits per weight, read as next-digit predictors on in-context streams of a declared class of 177,591 programs, put mass on continuations the class never produces, and none stays within the 3.32 bits by which a mixture carrying a Krichevsky-Trofimov code at weight 0.10 can trail that code: at twenty-four symbols the best trails it by 4.5 bits, the median by 36.5 and the worst by 128. Over 256 symbols the three checkpoints nearest that bound settle to 7-13 bits, as a mixture carrying the code at a smaller weight would, while others keep losing ground; and the reading closest to Solomonoff’s allocation tracks the mass the models put outside the alphabet, so neither scale nor post-training is shown to move it towards a program-weighted mixture. Frontier developers adopt neurosymbolic approaches, including Fable and Astra, incorporating model synthesis via neurosymbolic computation. They are no longer purely statistical LLMs, making them better, though still limited, candidates for higher forms of induction and model synthesis. Our results contradict recent literature suggesting LLMs are Solomonoff-induction estimators based on arguments related to purely statistical compression algorithms.

术语表

Algorithmic Information Theory (AIT)
算法信息论(AIT)
Machine Learning (ML)
机器学习(ML)
Artificial General Intelligence (AGI)
通用人工智能(AGI)
Large Language Model (LLM)
大型语言模型(LLM)
Solomonoff induction
Solomonoff 归纳
Solomonoff prior
Solomonoff 先验
algorithmic probability
算法概率
program-weighted universal mixture
程序加权通用混合分布
next-token prediction
下一词元预测
cross-entropy
交叉熵
negative log-likelihood
负对数似然
inductive principle
归纳原则
resource-bounded estimator
资源受限估计器
Krichevsky-Trofimov code
Krichevsky-Trofimov 编码
recursive self-training
递归自训练
recursive self-improvement
递归自我改进
model collapse
模型坍塌
model synthesis
模型合成
neurosymbolic computation
神经符号计算
Minimum Description Length (MDL)
最小描述长度(MDL)
Shannon entropy rate
Shannon 熵率
data-processing inequality
数据处理不等式
Levin’s algorithmic-information non-growth results
Levin 算法信息非增长结果
program synthesis
程序合成
Transformer
Transformer
AIXI
AIXI
JEPA
JEPA
Fable
Fable
Astra
Astra