Evaluating Large Language Models Trained on Code
Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman Alex Ray Raul Puri Gretchen Krueger Michael Petrov Heidy Khlaaf Girish Sastry Pamela Mishkin Brooke Chan Scott Gray Nick Ryder Mikhail Pavlov Alethea Power Lukasz Kaiser Mohammad Bavarian Clemens Winter Philippe Tillet Felipe Petroski Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes Ariel Herbert-Voss William Hebgen Guss Alex Nichol Alex Paino Nikolas Tezak Jie Tang Igor Babuschkin Suchir Balaji Shantanu Jain William Saunders Christopher Hesse Andrew N. Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Alec Radford Matthew Knight Miles Brundage Mira Murati Katie Mayer Peter Welinder Bob McGrew Dario Amodei Sam McCandlish Ilya Sutskever Wojciech Zaremba
Abstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
中文速览
用专门针对代码的大规模语言模型来自动生成正确的Python函数，一直面临缺乏可靠评估手段和模型能力不足两大瓶颈。研究团队将GPT模型在1590亿字节的GitHub公开代码上进行微调，得到Codex，并同步构建了包含164道编程题及单元测试的HumanEval基准，以"能否通过单元测试"这一功能正确性标准来衡量模型表现。结果显示，120亿参数的Codex单次采样可解决28.8%的题目，而若每题采样100次再从中挑选，正确率跃升至70.2%，远超GPT-3的近乎0%和GPT-J的11.4%。这项工作不仅推动了AI辅助编程工具（如GitHub Copilot）的落地，也揭示了以功能正确性替代BLEU分数作为代码生成评价指标的必要性，并对代码生成技术的安全与社会影响进行了系统讨论。
原文 arXiv:2107.03374;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2107.03374v2