Aha.
正在载入中英对照阅读…

arXiv:2606.17514 · 中英对照阅读

通过迭代反馈循环解锁大语言模型(LLM)的代码纠错能力

Unlocking LLM Code Correction with Iterative Feedback Loops

Le Zhang、Suresh Kothari

中文速览

让大语言模型写出代码后,能不能根据报错和测试结果自己一步步修好,是这项研究关注的问题。研究者让四种模型用 Python 和 Java 完成数百道编程题,再把编译错误、运行结果和测试反馈不断交给模型,观察最多十轮后的变化。结果显示,具备推理能力的模型通常越改越好,也更会利用反馈;语法和运行错误较容易修复,而逻辑错误和算法效率问题更难。研究说明,评估模型不能只看第一次写对多少,还要看它能否从失败中学会修正,这对打造更可靠的自动编程工具很重要。

摘要

大语言模型(LLM)在代码生成方面展现出卓越能力。然而,现有评估大多只关注单次尝试的准确率,忽视了在现实编程中至关重要的迭代优化过程。本研究系统考察了大语言模型(LLM)通过执行反馈纠正自身代码的能力。研究选取四种模型和两种主流编程语言中的真实编程问题,采用迭代优化框架评估模型表现;在每次尝试后,模型都会收到编译器错误信息和测试用例反馈。本研究提出了用于评估代码错误、分析纠错模式以及比较推理模型与非推理模型效果的指标,为理解大语言模型(LLM)驱动的代码生成系统中反馈循环的作用,并将其付诸实践,提供了切实可行的见解。结果表明,推理模型在迭代过程中持续改进,在利用反馈方面显著优于非推理模型;相比逻辑错误或算法错误,语法错误和运行时错误更易于纠正。

Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs’ ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.

术语表

Large Language Model (LLM)
大语言模型(LLM)
code generation
代码生成
code rectification
代码纠错
execution feedback
执行反馈
feedback loop
反馈循环
iterative refinement
迭代优化
pass@1
pass@1(单次通过率)
pass@k
pass@k(k次尝试通过率)
reasoning model
推理模型
non-reasoning model
非推理模型
compiler error message
编译器错误信息
testcase feedback
测试用例反馈
syntactic error
语法错误
runtime error
运行时错误
logical error
逻辑错误
algorithmic failure
算法错误
algorithmic optimization
算法优化
computational constraint
计算约束
time constraint
时间约束
space constraint
空间约束
Core Dataset
核心数据集
Strain Dataset
效率压力数据集
Challenge Dataset
挑战数据集
LeetCode
LeetCode
DeepSeek-R1
DeepSeek-R1
DeepSeek-V3
DeepSeek-V3
GPT-o4-mini
GPT-o4-mini
GPT-4.1-mini
GPT-4.1-mini