CoTexT: Multi-task Learning with Code-Text Transformer
Long Phan Case Western Reserve University, Ohio, USA Hieu Tran University of Science, VNU-HCM, Vietnam Daniel Le Case Western Reserve University, Ohio, USA Hieu Nguyen Case Western Reserve University, Ohio, USA James Anibal Case Western Reserve University, Ohio, USA Alec Peltekian Case Western Reserve University, Ohio, USA Yanfang Ye Case Western Reserve University, Ohio, USA
Abstract
We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging. We train CoTexT on different combinations of available PL corpus including both ”bimodal” and ”unimodal” data. Here, bimodal data is the combination of text and corresponding code snippets, whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models.
中文速览
如何让机器同时"看懂"自然语言和代码，并胜任代码摘要、代码生成、缺陷检测、代码修复等多种任务，是当前软件智能化研究的核心挑战。研究者提出了 CoTexT（Code and Text Transfer Transformer），一个基于 T5 编解码器架构、在大规模编程语言语料上做自监督预训练的模型，训练数据同时包含"双模态"数据（自然语言描述与对应代码片段）和"单模态"数据（纯代码片段），并针对编程语言中特殊符号词表缺失的问题做了专门处理。在 CodeXGLUE 基准上，CoTexT 在代码摘要（6 种编程语言）、代码生成、代码修复（小型和中型数据集）以及缺陷检测四项任务上均刷新了当时的最优结果，超越了 CodeBERT、PLBART 等已有模型。这项工作表明，将通用预训练语言模型迁移并适配到编程语言领域，结合多任务学习框架，能够显著提升代码智能任务的性能，同时研究者公开了预训练权重和代码，为后续研究提供了直接可用的基础。
原文 arXiv:2105.08645;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.08645v4