An Analysis of the Automatic Bug Fixing Performance of ChatGPT
Dominik Sobania Carol Hanna Johannes Gutenberg University Mainz Email: University College London Email: Martin Briesch Justyna Petke Johannes Gutenberg University Mainz Email: University College London Email:
Abstract
To support software developers in finding and fixing software bugs, several automated program repair techniques have been introduced. Given a test suite, standard methods usually either synthesize a repair, or navigate a search space of software edits to find test-suite passing variants. Recent program repair methods are based on deep learning approaches. One of these novel methods, which is not primarily intended for automated program repair, but is still suitable for it, is ChatGPT. The bug fixing performance of ChatGPT, however, is so far unclear. Therefore, in this paper we evaluate ChatGPT on the standard bug fixing benchmark set, QuixBugs, and compare the performance with the results of several other approaches reported in the literature. We find that ChatGPT’s bug fixing performance is competitive to the common deep learning approaches CoCoNut and Codex and notably better than the results reported for the standard program repair approaches. In contrast to previous approaches, ChatGPT offers a dialogue system through which further information, e.g., the expected output for a certain input or an observed error message, can be entered. By providing such hints to ChatGPT, its su
中文速览
自动修复软件缺陷（Automated Program Repair，APR）一直是软件工程领域的难题，现有工具要么依赖测试套件搜索补丁，要么用深度学习模型翻译式地生成修复，各有局限。研究者将目光投向ChatGPT，在标准缺陷修复基准集QuixBugs的40个Python程序上系统评测了它的修复能力，并与传统APR工具及CoCoNut、Codex等深度学习方法横向比较。结果显示，仅凭一次提问，ChatGPT就能修复其中19个缺陷，与CoCoNut（19个）和Codex（21个）旗鼓相当，远超传统方法（仅7个）；更关键的是，充分利用ChatGPT的对话功能、向其补充一条失败测试用例或错误信息作为提示后，修复数量跃升至31个，超越了当时所有已报告方法的最优水平。这一发现表明，对话式交互是提升AI辅助程序修复能力的重要杠杆，具有很高的实用价值。
原文 arXiv:2301.08653;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.08653v1