Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesis

摘要

We connect two questions in Algorithmic Information Theory (AIT), Machine Learning (ML) and Artificial General Intelligence (AGI): whether LLMs estimate Solomonoff induction, and whether they can self-improve towards an AI Singularity. We provide theoretical, methodological and empirical answers in the negative but show how limits can be circumvented. Cross-entropy, negative log-likelihood and related next-token objectives cannot alone implement Solomonoff induction: they fit supplied conditionals rather than a program-weighted universal mixture. More computation can improve fit within a fixed objective but cannot change its inductive principle without external hyperparameter or architectural tuning; they alone do not deliver Solomonoff-Levin optimal prediction. For finite learners and observers, theoretical boundaries become less decisive and approaches diverge. Resource-bounded estimators are finite mechanism-search tools whose divergence does not violate algorithmic information conservation. Twenty-six served language-model checkpoints across five pre-training families, 0.8-35 billion parameters and 1.9-8.5 bits per weight, read as next-digit predictors on in-context streams of a declared class of 177,591 programs, put mass on continuations the class never produces, and none stays within the 3.32 bits by which a mixture carrying a Krichevsky-Trofimov code at weight 0.10 can trail that code: at twenty-four symbols the best trails it by 4.5 bits, the median by 36.5 and the worst by 128. Over 256 symbols the three checkpoints nearest that bound settle to 7-13 bits, as a mixture carrying the code at a smaller weight would, while others keep losing ground; and the reading closest to Solomonoff’s allocation tracks the mass the models put outside the alphabet, so neither scale nor post-training is shown to move it towards a program-weighted mixture. Frontier developers adopt neurosymbolic approaches, including Fable and Astra, incorporating model synthesis via neurosymbolic computation. They are no longer purely statistical LLMs, making them better, though still limited, candidates for higher forms of induction and model synthesis. Our results contradict recent literature suggesting LLMs are Solomonoff-induction estimators based on arguments related to purely statistical compression algorithms.