Fault-Aware Neural Code Rankers
Jeevana Priya Inala、Chenglong Wang、Mei Yang、Andres Codas Mark Encarnación、Shuvendu K Lahiri、Madanlal Musuvathi、Jianfeng Gao、Microsoft Research {jinala,chenwang,meiyang,andres.codas,markenc,
Abstract
Large language models (LLMs) have demonstrated an impressive ability to generate code for various programming tasks. In many instances, LLMs can generate a correct program for a task when given numerous trials. Consequently, a recent trend is to do large scale sampling of programs using a model and then filtering/ranking the programs based on the program execution on a small number of known unit tests to select one candidate solution. However, these approaches assume that the unit tests are given and assume the ability to safely execute the generated programs (which can do arbitrary dangerous operations such as file manipulations). Both of the above assumptions are impractical in real-world software development. In this paper, we propose CodeRanker, a neural ranker that can predict the correctness of a sampled program without executing it. Our CodeRanker is fault-aware i.e., it is trained to predict different kinds of execution information such as predicting the exact compile/runtime error type (e.g., an IndexError or a TypeError). We show that CodeRanker can significantly increase the pass@1 accuracy of various code generation models (including Codex chen2021evaluating , GPT-Neo,
中文速览
大规模采样再筛选已成为提升代码生成质量的主流思路，但现有方法依赖真实运行代码并要求用户提供单元测试，在实际开发场景中既不安全又难以落地。为此，研究者提出了 CodeRanker——一个无需执行代码就能预测程序正确性的神经排序器（neural ranker），其核心创新在于"故障感知"训练：通过让模型预测程序会触发哪类具体错误（如 IndexError、TypeError，或输出类型/长度不符等），而非仅判断对错，迫使模型更深入地理解程序逻辑与任务意图。实验表明，CodeRanker 能将 Codex 在 APPS 数据集验证集上的 pass@1 准确率从 26% 提升至 39.6%，在 HumanEval 和 MBPP 上也分别提升约 6 个百分点，对 GPT-Neo、GPT-J 等开源模型同样有效，且无需在目标数据集上重新训练即可迁移使用。这项工作证明，通过学习细粒度的程序失败模式，神经排序器完全可以在不执行任何代码的前提下大幅缩小"采样一次"与"采样百次"之间的性能鸿沟，为真实软件开发场景提供了一条更安全、更实用的代码排序路径。
原文 arXiv:2206.03865;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.03865v2