arXiv:2606.17514 · 中英对照阅读
通过迭代反馈循环解锁大语言模型(LLM)的代码纠错能力
Unlocking LLM Code Correction with Iterative Feedback Loops
中文速览
让大语言模型写出代码后,能不能根据报错和测试结果自己一步步修好,是这项研究关注的问题。研究者让四种模型用 Python 和 Java 完成数百道编程题,再把编译错误、运行结果和测试反馈不断交给模型,观察最多十轮后的变化。结果显示,具备推理能力的模型通常越改越好,也更会利用反馈;语法和运行错误较容易修复,而逻辑错误和算法效率问题更难。研究说明,评估模型不能只看第一次写对多少,还要看它能否从失败中学会修正,这对打造更可靠的自动编程工具很重要。
摘要
大语言模型(LLM)在代码生成方面展现出卓越能力。然而,现有评估大多只关注单次尝试的准确率,忽视了在现实编程中至关重要的迭代优化过程。本研究系统考察了大语言模型(LLM)通过执行反馈纠正自身代码的能力。研究选取四种模型和两种主流编程语言中的真实编程问题,采用迭代优化框架评估模型表现;在每次尝试后,模型都会收到编译器错误信息和测试用例反馈。本研究提出了用于评估代码错误、分析纠错模式以及比较推理模型与非推理模型效果的指标,为理解大语言模型(LLM)驱动的代码生成系统中反馈循环的作用,并将其付诸实践,提供了切实可行的见解。结果表明,推理模型在迭代过程中持续改进,在利用反馈方面显著优于非推理模型;相比逻辑错误或算法错误,语法错误和运行时错误更易于纠正。
Large Language Models have shown remarkable capabilities in code generation. However, most existing evaluations focus only on single-attempt accuracy and overlook the iterative refinement process that is central to real-world programming. This study presents a systematic investigation of LLMs’ ability to rectify their own code through execution feedback. Using real-world programming problems across four models and two major programming languages, this study evaluates performance using iterative refinement framework where LLMs receive compiler error messages and testcase feedback after each attempt. This study introduces metrics to evaluate code failures, analyze rectification patterns, and compare the effectiveness of reasoning and non-reasoning models, offering actionable insights into both the understanding and practical application of feedback loops in LLM-driven code generation systems. Results show that reasoning models consistently improve over iterations, substantially outperforming non-reasoning models in leveraging feedback, while syntactic and runtime errors are far more tractable than logical or algorithmic failures.
术语表
- Large Language Model (LLM)
- 大语言模型(LLM)
- code generation
- 代码生成
- code rectification
- 代码纠错
- execution feedback
- 执行反馈
- feedback loop
- 反馈循环
- iterative refinement
- 迭代优化
- pass@1
- pass@1(单次通过率)
- pass@k
- pass@k(k次尝试通过率)
- reasoning model
- 推理模型
- non-reasoning model
- 非推理模型
- compiler error message
- 编译器错误信息
- testcase feedback
- 测试用例反馈
- syntactic error
- 语法错误
- runtime error
- 运行时错误
- logical error
- 逻辑错误
- algorithmic failure
- 算法错误
- algorithmic optimization
- 算法优化
- computational constraint
- 计算约束
- time constraint
- 时间约束
- space constraint
- 空间约束
- Core Dataset
- 核心数据集
- Strain Dataset
- 效率压力数据集
- Challenge Dataset
- 挑战数据集
- LeetCode
- LeetCode
- DeepSeek-R1
- DeepSeek-R1
- DeepSeek-V3
- DeepSeek-V3
- GPT-o4-mini
- GPT-o4-mini
- GPT-4.1-mini
- GPT-4.1-mini