Falsification, Not Exposure: An Internally Preregistered Placebo-Controlled Decomposition of Self-Repair Feedback in Frozen Small Code Models

摘要

In deployment settings where retraining is not feasible, small frozen code models are commonly asked to repair a failed program after being shown their own failing output. This practice is often treated as an additional retry mechanism. From a Popperian perspective, however, it raises a deeper epistemic and measurement problem. A generated program can be viewed as a tentative conjecture, while a violation observed during test execution can be viewed as a test-oracle-relative and executable counterexample. The scientific value of feedback should therefore be attributed not to re-exposure to failing code itself, but to whether the candidate conjecture is opened to external, executable, and auditable criticism. Prior work has often bundled failing code, execution evidence, and retry instruction into a single feedback packet, without decomposing packet components against a blind resampling baseline under the same output-generation budget and against content-free placebo controls. As the third stage of a falsification-centered measurement program, this study presents a placebo-controlled evaluation instrument that compares bare failing code, executed facts, code-plus-facts, neither, and template- and bullet-count-matched placebo conditions on the same task–model units. The contribution is not a new repair algorithm, but a reflexive measurement methodology combining packet decomposition, placebo mirroring, matched-budget comparison, same-unit discordant-pair evaluation, fresh-generation confirmation, and provenance-based executable audits, so that both the model’s program conjecture and the researcher’s claim that “feedback content works” are made falsifiable. Across six HumanEval+ and MBPP+ benchmark $\times$ model cells with three 0.5B–1.5B frozen code models, 290 dead task–cell units were evaluated, where no candidate in a cached best-of- $N$ pool ( $N=8$ ) passed the public tier. The main run produced $7{,}000$ fresh seeded generations, and a preregistered diagnostic follow-up produced $1{,}400$ additional generations. True unlock required passing both the public and prompt-hidden EvalPlus tiers. Blind resampling exceeded bare-code retry by $+18$ net unlocks (discordant $25/7$ , Holm-adjusted $p=0.0021$ ). Code-plus-facts produced $+18$ net recovery relative to bare code (discordant $21/3$ , Holm-adjusted $p=0.00042$ ), and a $+15$ advantage over the same-code generic-bullet placebo supported the contribution of executed fact content (Holm-adjusted $p=0.0041$ ). In the diagnostic follow-up, an instruction-only effect was not distinguishable ( $+3$ , $p=0.36$ ), which is not evidence of absence. The remaining difference was associated with code-bearing context, although pure code causation was not claimed because code content remained confounded with prompt length. Code-plus-facts and blind resampling tied at the pooled level with $26$ unlocks each, which is not evidence of equivalence, non-inferiority, non-superiority, or unit-level identical behavior, but only zero observed pooled difference in the tested sample. The content-attribution contrast changed direction in a small disjoint sensitivity sample, and surface-form explanations were not fully ruled out. Six prospective external-controller follow-ups on the search-bound stratum further supported this boundary: the two strongest content channels, own near-miss code and runtime execution trace, tied with content-free shape placebo. The findings are limited to observed unlock outcomes and make no direct claim about latent sampling capacity, model distribution, or general program-synthesis ability. Taken together, falsification in the tested frozen-small-model regime was valuable not as Popperian vocabulary, verbal self-critique, or re-exposure to failing code, but as comparison with externally produced, oracle-relative, executable counterexamples. This execution-grounded criticism was observed as a limited signal repairing harmful code-bearing conditioning, not as a content lever exceeding blind resampling. The transferable contribution is therefore a methodology for making self-repair feedback claims falsifiable through content-vs-form, discovery-vs-confirmation, and audit-vs-statistical-verdict distinctions.