InCoder: A Generative Model for Code Infilling and Synthesis
Daniel Fried∗♡†♢ Armen Aghajanyan∗♡ Jessy Lin♣ Sida Wang♡ Eric Wallace♣ Freda Shi△ Ruiqi Zhong♣ Wen-tau Yih♡ Luke Zettlemoyer♡† Mike Lewis♡ Facebook AI Research♡ University of Washington† UC Berkeley♣ TTI-Chicago△ Carnegie Mellon University♢ Equal contribution
Abstract
Code is seldom written in a single left-to-right pass and is instead repeatedly edited and refined. We introduce InCoder, a unified generative model that can perform program synthesis (via left-to-right generation) as well as editing (via masking and infilling). InCoder is trained to generate code files from a large corpus of permissively licensed code, where regions of code have been randomly masked and moved to the end of each file, allowing code infilling with bidirectional context. Our model is the first large generative code model that is able to infill arbitrary regions of code, which we evaluate in a zero-shot setting on challenging tasks such as type inference, comment generation, and variable re-naming. We find that the ability to condition on bidirectional context substantially improves performance on these tasks, while still performing comparably on standard program synthesis benchmarks in comparison to left-to-right only models pretrained at similar scale. Our models and code are publicly released.111https://sites.google.com/view/incoder-code-models/
中文速览
代码在实际开发中从来不是从左到右一次写完的，而是需要反复修改和填补——但现有的大型代码生成模型只能从左向右生成，无法利用右侧上下文来"填空"。为此，研究者提出了 InCoder，通过一种"因果掩码（causal masking）"训练目标，在训练时随机遮住代码片段并将其移到文件末尾，让模型同时学会从左到右生成和利用双向上下文填充任意位置的代码。在类型推断、注释生成、变量重命名、代码行填充等零样本测试中，InCoder 的填充能力大幅超越只能从左向右生成的同类模型，在部分任务上甚至接近经过专项微调的最先进方法，同时在标准代码合成基准上也保持了与同规模单向模型相当的表现。这项工作的意义在于：它首次将"生成"与"编辑"统一到同一个大模型中，为代码补全、缺陷修复等真实开发场景提供了更实用的基础模型，且模型与代码均已公开发布。
原文 arXiv:2204.05999;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.05999v3