Competition-Level Code Generation with AlphaCode
Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser Rémi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago Thomas Hubert Peter Choy Cyprien de Masson d’Autume Igor Babuschkin Xinyun Chen Po-Sen Huang Johannes Welbl Sven Gowal Alexey Cherepanov James Molloy Daniel J. Mankowitz Esme Sutherland Robson Pushmeet Kohli Nando de Freitas Koray Kavukcuoglu Oriol Vinyals
Abstract
Programming is a powerful and ubiquitous problem-solving tool. Developing systems that can assist programmers or even generate programs independently could make programming more productive and accessible, yet so far incorporating innovations in AI has proven challenging. Recent large-scale language models have demonstrated an impressive ability to generate code, and are now able to complete simple programming tasks. However, these models still perform poorly when evaluated on more complex, unseen problems that require problem-solving skills beyond simply translating instructions into code. For example, competitive programming problems which require an understanding of algorithms and complex natural language remain extremely challenging. To address this gap, we introduce AlphaCode, a system for code generation that can create novel solutions to these problems that require deeper reasoning. In simulated evaluations on recent programming competitions on the Codeforces platform, AlphaCode achieved on average a ranking of top 54.3% in competitions with more than 5,000 participants. We found that three key components were critical to achieve good and reliable performance: (1) an extensiv
中文速览
竞技编程长期被视为检验AI推理能力的高难度标杆，现有代码生成模型在这类需要深度算法理解的复杂问题上求解率极低，且评测方法也存在大量误报。DeepMind提出的AlphaCode系统通过三个关键设计来应对这一挑战：在精心清洗的竞技编程数据集上预训练和微调大规模Transformer模型、大批量采样候选程序，再利用题目自带的样例测试过滤并聚类，最终缩减为少量提交候选。在Codeforces平台10场真实比赛的模拟评测中，AlphaCode平均排名位于参赛选手前54.3%，折合等级分约1238分，跻身近半年活跃用户的前28%，是历史上首个在编程竞赛中达到如此水准的AI系统。这一成果表明，结合大规模语言模型与系统化搜索-过滤策略，AI已能真正"理解问题、设计算法、生成完整代码"，而不仅仅是翻译简单指令，为未来更通用的程序合成和AI辅助编程奠定了重要基础。
原文 arXiv:2203.07814;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.07814v1