arXiv:2606.31511 · 中英对照阅读
证伪,而非暴露:对冻结小型代码模型中自我修复反馈进行内部预注册的安慰剂对照分解
Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models
中文速览
小型冻结代码模型修不好程序时,关键问题是:把失败代码和执行结果喂回去,是否真比同样预算下重新抽样更有效。研究者在多种反馈组合之间做了对照,并用模板和项目符号匹配的“安慰剂”排除只是提示格式不同带来的影响,再让新生成结果接受公开与隐藏测试。结果显示,盲目重抽样比只重看失败代码更容易修好程序;加入真实执行事实也比只给失败代码或泛泛提示更有帮助,但整体并没有胜过盲目重抽样,且这些发现只适用于测试的模型和任务范围。这项研究的重要之处,不是提出新的修复技巧,而是提供一套更严谨的检验方法,帮助分清模型究竟受到了真实反例的帮助,还是只是被额外文字和提示形式影响了。
摘要
在无法进行再训练的部署环境中,小型冻结代码模型通常会在看到自身失败输出后,被要求修复失败的程序。这种做法通常被视为一种额外的重试机制。然而,从波普尔式视角看,它提出了更深层的认识论与测量问题。生成的程序可被视为暂定的猜想,而测试执行期间观察到的违规则可被视为相对于测试预言机且可执行的反例。因此,反馈的科学价值不应归因于再次呈现失败代码本身,而应取决于候选猜想是否接受外部、可执行且可审计的批评。以往工作往往将失败代码、执行证据和重试指令捆绑成一个反馈包,却没有在相同输出生成预算下,将包内各组成部分与盲重采样基线及不含内容的安慰剂对照进行拆分比较。作为以证伪为核心的测量计划的第三阶段,本研究提出一种安慰剂对照评估工具,在相同的任务—模型单位上比较纯失败代码、已执行事实、代码加事实、两者皆无,以及模板和项目符号数量匹配的安慰剂条件。本研究的贡献并非一种新的修复算法,而是一套反身性测量方法,结合了反馈包拆分、安慰剂镜像、预算匹配比较、同单位不一致配对评估、新生成确认和基于溯源的可执行审计,从而使模型的程序猜想以及研究者关于“反馈内容有效”的主张都可被证伪。在由三个参数量为 0.5B–1.5B 的冻结代码模型构成的六个 HumanEval+ 与 MBPP+ 基准测试 × 模型单元中,本研究评估了 290 个无解锁任务—单元,即缓存的 best-of-N 候选池($N=8$)中没有任何候选通过公开层级。主实验生成了 $7{,}000$ 个新的随机种子生成样本,预注册的诊断性后续实验又生成了 $1{,}400$ 个样本。真正的解锁要求同时通过公开层级和提示隐藏层级的 EvalPlus。盲重采样比纯代码重试多实现了 $+18$ 次净解锁(不一致配对为 $25/7$,经 Holm 校正的 $p=0.0021$)。与纯代码相比,代码加事实带来了 $+18$ 次净恢复(不一致配对为 $21/3$,经 Holm 校正的 $p=0.00042$);与代码相同、仅使用通用项目符号的安慰剂相比,其优势为 $+15$,支持已执行事实内容的贡献(经 Holm 校正的 $p=0.0041$)。在诊断性后续实验中,无法区分仅指令带来的效应($+3$,$p=0.36$),但这并非不存在效应的证据。其余差异与包含代码的上下文相关,但由于代码内容仍与提示长度混杂,本研究并未主张这是纯粹由代码导致的因果效应。在汇总层面,代码加事实与盲重采样各解锁了 $26$ 个任务;这并非等效性、非劣效性、非优效性或单位层面行为完全相同的证据,只表明在所测试样本中观察到的汇总差异为零。在一个较小且互不重叠的敏感性样本中,内容归因对比的方向发生了变化,且未能完全排除表层形式解释。针对搜索受限层级开展的六项前瞻性外部控制器后续实验进一步支持了这一边界:两个最强的内容通道——模型自身的近失误代码和运行时执行轨迹——与不含内容的形式安慰剂效果持平。研究结果仅限于观察到的解锁结果,不对潜在采样能力、模型分布或一般程序合成能力作出直接主张。总而言之,在本研究所测试的冻结小模型情境中,证伪的价值不在于使用波普尔式术语、进行言语上的自我批评或再次呈现失败代码,而在于将其与外部产生的、相对于测试预言机且可执行的反例进行比较。研究观察到,这种以执行为依据的批评是一种有限的信号,能够修复有害的代码条件化效应,但并非优于盲重采样的内容杠杆。因此,可迁移的贡献在于提供一种方法论,通过区分内容与形式、发现与确认、审计与统计判决,使关于自我修复反馈的主张能够被证伪。
In deployment settings where retraining is not feasible, small frozen code models are commonly asked to repair a failed program after being shown their own failing output. This practice is often treated as an additional retry mechanism. From a Popperian perspective, however, it raises a deeper epistemic and measurement problem. A generated program can be viewed as a tentative conjecture, while a violation observed during test execution can be viewed as a test-oracle-relative and executable counterexample. The scientific value of feedback should therefore be attributed not to re-exposure to failing code itself, but to whether the candidate conjecture is opened to external, executable, and auditable criticism. Prior work has often bundled failing code, execution evidence, and retry instruction into a single feedback packet, without decomposing packet components against a blind resampling baseline under the same output-generation budget and against content-free placebo controls. As the third stage of a falsification-centered measurement program, this study presents a placebo-controlled evaluation instrument that compares bare failing code, executed facts, code-plus-facts, neither, and template- and bullet-count-matched placebo conditions on the same task–model units. The contribution is not a new repair algorithm, but a reflexive measurement methodology combining packet decomposition, placebo mirroring, matched-budget comparison, same-unit discordant-pair evaluation, fresh-generation confirmation, and provenance-based executable audits, so that both the model’s program conjecture and the researcher’s claim that “feedback content works” are made falsifiable. Across six HumanEval+ and MBPP+ benchmark $\times$ model cells with three 0.5B–1.5B frozen code models, 290 dead task–cell units were evaluated, where no candidate in a cached best-of- $N$ pool ( $N=8$ ) passed the public tier. The main run produced $7{,}000$ fresh seeded generations, and a preregistered diagnostic follow-up produced $1{,}400$ additional generations. True unlock required passing both the public and prompt-hidden EvalPlus tiers. Blind resampling exceeded bare-code retry by $+18$ net unlocks (discordant $25/7$ , Holm-adjusted $p=0.0021$ ). Code-plus-facts produced $+18$ net recovery relative to bare code (discordant $21/3$ , Holm-adjusted $p=0.00042$ ), and a $+15$ advantage over the same-code generic-bullet placebo supported the contribution of executed fact content (Holm-adjusted $p=0.0041$ ). In the diagnostic follow-up, an instruction-only effect was not distinguishable ( $+3$ , $p=0.36$ ), which is not evidence of absence. The remaining difference was associated with code-bearing context, although pure code causation was not claimed because code content remained confounded with prompt length. Code-plus-facts and blind resampling tied at the pooled level with $26$ unlocks each, which is not evidence of equivalence, non-inferiority, non-superiority, or unit-level identical behavior, but only zero observed pooled difference in the tested sample. The content-attribution contrast changed direction in a small disjoint sensitivity sample, and surface-form explanations were not fully ruled out. Six prospective external-controller follow-ups on the search-bound stratum further supported this boundary: the two strongest content channels, own near-miss code and runtime execution trace, tied with content-free shape placebo. The findings are limited to observed unlock outcomes and make no direct claim about latent sampling capacity, model distribution, or general program-synthesis ability. Taken together, falsification in the tested frozen-small-model regime was valuable not as Popperian vocabulary, verbal self-critique, or re-exposure to failing code, but as comparison with externally produced, oracle-relative, executable counterexamples. This execution-grounded criticism was observed as a limited signal repairing harmful code-bearing conditioning, not as a content lever exceeding blind resampling. The transferable contribution is therefore a methodology for making self-repair feedback claims falsifiable through content-vs-form, discovery-vs-confirmation, and audit-vs-statistical-verdict distinctions.
术语表
- frozen code models
- 冻结代码模型
- self-repair
- 自我修复
- execution feedback
- 执行反馈
- retry loop
- 重试循环
- Popperian perspective
- 波普尔式视角
- conjecture
- 猜想
- falsification
- 证伪
- executable counterexample
- 可执行反例
- test oracle
- 测试预言机
- placebo control
- 安慰剂对照
- blind resampling
- 盲重采样
- matched-budget comparison
- 预算匹配比较
- same-unit discordant-pair evaluation
- 同单位不一致配对评估
- provenance-based executable audit
- 基于溯源的可执行审计
- fresh-generation confirmation
- 新生成确认
- HumanEval+
- HumanEval+
- MBPP+
- MBPP+
- EvalPlus
- EvalPlus
- public tier
- 公开层级
- prompt-hidden tier
- 提示隐藏层级
- best-of-N
- best-of-N
- unlock
- 解锁
- content attribution
- 内容归因
- surface-form explanation
- 表层形式解释
- external controller
- 外部控制器
- runtime execution trace
- 运行时执行轨迹
- program synthesis
- 程序合成
- large language model (LLM)
- 大语言模型(LLM)
- small language model
- 小语言模型