MASS: Masked Sequence to Sequence Pre-training for Language Generation
Kaitao Song Xu Tan Tao Qin Jianfeng Lu Tie-Yan Liu
Abstract
Pre-training and fine-tuning, e.g., BERT (Devlin et al., 2018), have achieved great success in language understanding by transferring knowledge from rich-resource pre-training task to the low/zero-resource downstream tasks. Inspired by the success of BERT, we propose MAsked Sequence to Sequence pre-training (MASS) for encoder-decoder based language generation. MASS adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence: its encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and its decoder tries to predict this masked fragment. In this way, MASS can jointly train the encoder and decoder to develop the capability of representation extraction and language modeling. By further fine-tuning on a variety of zero/low-resource language generation tasks, including neural machine translation, text summarization and conversational response generation (3 tasks and totally 8 datasets), MASS achieves significant improvements over baselines without pre-training or with other pre-training methods. Specially, we achieve state-of-the-art accuracy (37.5 in terms of BLEU score) on the unsupervised Engl
中文速览
语言生成任务（机器翻译、文本摘要、对话回复等）普遍面临标注数据稀缺的难题，而现有预训练方法（如BERT）只针对编码器设计，无法直接迁移到需要编码器-解码器协同工作的生成场景。为此，微软研究院提出了MASS（掩码序列到序列预训练），其核心思路是：在编码器端随机遮住句子中一段连续词块，让解码器去预测这段被遮住的内容，由此迫使编码器理解上下文语义、同时逼着解码器更多依赖编码器表示而非自身历史输出，从而实现编解码器的联合预训练。在无监督英法翻译上，MASS取得了37.5的BLEU分数，不仅超越此前最优无监督方法逾4个点，甚至超过了早期有监督的注意力机制翻译模型；在低资源机器翻译、文本摘要和对话生成等共8个数据集上也全面领先基线。这项工作的意义在于，它填补了预训练技术在生成任务上的空白，为数据匮乏的序列生成问题提供了一套通用且有效的解决方案。
原文 arXiv:1905.02450;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.02450v5