CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, Steven C.H. Hoi Salesforce Research https://github.com/salesforce/CodeRL Equal contribution.Corresponding authors: {hungle,
Abstract
Program synthesis or code generation aims to generate a program that satisfies a problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised fine-tuning procedure to train a code generation model only from the pairs of natural-language problem descriptions and ground-truth programs. Such paradigm largely ignores some important but potentially useful signals in the problem specification such as unit tests, which thus often results in poor performance when solving complex unseen coding tasks. To address the limitations, we propose “CodeRL”, a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL). Specifically, during training, we treat the code-generating LM as an actor network, and introduce a critic network that is trained to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, we introduce a new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs bas
中文速览
大规模预训练语言模型在代码生成上虽然取得了不错的成绩，但面对竞赛级别的复杂编程题时仍然力不从心，根本原因在于训练时只靠"下一个词预测"的监督信号，完全忽略了单元测试所能提供的功能正确性反馈。为此，研究者提出 CodeRL 框架，把代码生成模型当作强化学习（Reinforcement Learning, RL）中的"演员"，同时训练一个"评论家"网络来预测生成程序能否通过单元测试，并将其反馈的密集信号反向传播给演员以优化生成质量；推理阶段还引入了一套"关键采样"策略，让模型能依据示例单元测试的运行结果和评论家评分，自动对失败程序进行修复或重新生成。在此基础上，研究者还扩展了 CodeT5 的预训练方案，增大了模型规模并补充了更丰富的预训练数据。实验结果显示，CodeRL 在高难度 APPS 基准上刷新了最优成绩，并在更简单的 MBPP 基准上展现出强劲的零样本迁移能力，证明让模型真正"看懂"程序对不对比单纯模仿人类写法要有效得多。
原文 arXiv:2207.01780;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2207.01780v3