Evaluating Large Language Models Trained on Code
Mark Chen Affiliation: OpenAI, San Francisco, California, USA. Correspondence to: Jerry Tworek Affiliation: OpenAI, San Francisco, California, USA. Correspondence to: Heewoo Jun Affiliation: OpenAI, San Francisco, California, USA. Correspondence to: Qiming Yuan Affiliation: OpenAI, San Francisco, California, USA. Correspondence to: Henrique Ponde de Oliveira Pinto Affiliation: OpenAI, San Francisco, California, USA. Jared Kaplan Affiliation: Anthropic AI, San Francisco, California, USA. Work performed while at OpenAI. Harri Edwards Affiliation: OpenAI, San Francisco, California, USA. Yuri Burda Affiliation: OpenAI, San Francisco, California, USA. Nicholas Joseph Affiliation: Anthropic AI, San Francisco, California, USA. Work performed while at OpenAI. Greg Brockman Affiliation: OpenAI, San Francisco, California, USA. Alex Ray Affiliation: OpenAI, San Francisco, California, USA. Raul Puri Affiliation: OpenAI, San Francisco, California, USA. Gretchen Krueger Affiliation: OpenAI, San Francisco, California, USA. Michael Petrov Affiliation: OpenAI, San Francisco, California, USA. Heidy Khlaaf Affiliation: Zipline, South San Francisco, California, USA. Work performed while at OpenAI Girish Sastry Affiliation: OpenAI, San Francisco, California, USA. Pamela Mishkin Affiliation: OpenAI, San Francisco, California, USA. Brooke Chan Affiliation: OpenAI, San Francisco, California, USA. Scott Gray Affiliation: OpenAI, San Francisco, California, USA. Nick Ryder Affiliation: OpenAI, San Francisco, California, USA. Mikhail Pavlov Affiliation: OpenAI, San Francisco, California, USA. Alethea Power Affiliation: OpenAI, San Francisco, California, USA. Lukasz Kaiser Affiliation: OpenAI, San Francisco, California, USA. Mohammad Bavarian Affiliation: OpenAI, San Francisco, California, USA. Clemens Winter Affiliation: OpenAI, San Francisco, California, USA. Philippe Tillet Affiliation: OpenAI, San Francisco, California, USA. Felipe Petroski Such Affiliation: OpenAI, San Francisco, California, USA. Dave Cummings Affiliation: OpenAI, San Francisco, California, USA. Matthias Plappert Affiliation: OpenAI, San Francisco, California, USA. Fotios Chantzis Affiliation: OpenAI, San Francisco, California, USA. Elizabeth Barnes Affiliation: OpenAI, San Francisco, California, USA. Ariel Herbert-Voss Affiliation: OpenAI, San Francisco, California, USA. William Hebgen Guss Affiliation: OpenAI, San Francisco, California, USA. Alex Nichol Affiliation: OpenAI, San Francisco, California, USA. Alex Paino Affiliation: OpenAI, San Francisco, California, USA. Nikolas Tezak Affiliation: OpenAI, San Francisco, California, USA. Jie Tang Affiliation: OpenAI, San Francisco, California, USA. Igor Babuschkin Affiliation: OpenAI, San Francisco, California, USA. Suchir Balaji Affiliation: OpenAI, San Francisco, California, USA. Shantanu Jain Affiliation: OpenAI, San Francisco, California, USA. William Saunders Affiliation: OpenAI, San Francisco, California, USA. Christopher Hesse Affiliation: OpenAI, San Francisco, California, USA. Andrew N. Carr Affiliation: OpenAI, San Francisco, California, USA. Jan Leike Affiliation: OpenAI, San Francisco, California, USA. Josh Achiam Affiliation: OpenAI, San Francisco, California, USA. Vedant Misra Affiliation: OpenAI, San Francisco, California, USA. Evan Morikawa Affiliation: OpenAI, San Francisco, California, USA. Alec Radford Affiliation: OpenAI, San Francisco, California, USA. Matthew Knight Affiliation: OpenAI, San Francisco, California, USA. Miles Brundage Affiliation: OpenAI, San Francisco, California, USA. Mira Murati Affiliation: OpenAI, San Francisco, California, USA. Katie Mayer Affiliation: OpenAI, San Francisco, California, USA. Peter Welinder Affiliation: OpenAI, San Francisco, California, USA. Bob McGrew Affiliation: OpenAI, San Francisco, California, USA. Dario Amodei Affiliation: Anthropic AI, San Francisco, California, USA. Work performed while at OpenAI. Sam McCandlish Affiliation: Anthropic AI, San Francisco, California, USA. Work performed while at OpenAI. Ilya Sutskever Affiliation: OpenAI, San Francisco, California, USA. Wojciech Zaremba Affiliation: OpenAI, San Francisco, California, USA.
Abstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
原文 arXiv:2107.03374;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2107.03374v2