Cross-Lingual Natural Language Generation via Pre-Training
Zewen Chi†, Li Dong‡‡\ddagger, Furu Wei‡‡\ddagger, Wenhui Wang‡‡\ddagger, Xian-Ling Mao††{\dagger}, Heyan Huang††{\dagger} ††{\dagger}Beijing Institute of Technology ‡‡\ddaggerMicrosoft Research Contribution during internship at Microsoft Research.
Abstract
In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single language can be directly evaluated beyond that language (i.e., accepting multi-lingual input and producing multi-lingual output). Experimental results on question generation and abstractive summarization show that our model outperforms the machine-translation-based pipeline methods for zero-shot cross-lingual generation. Moreover, cross-lingual transfer improves NLG performance of low-resource languages by leveraging rich-resource language data. Our implementation and data are available at https://github.com/CZWin32768/xnlg.
中文速览
自然语言生成（NLG）任务的标注数据大多只有英语，导致模型难以直接迁移到其他语言。为此，研究者提出了跨语言预训练模型 XNLG，通过对序列到序列（sequence-to-sequence）模型的编码器和解码器同时进行单语和跨语言预训练，将多语言文本映射到共享的语义空间中。在仅用单一语言（如英语）的标注数据微调后，该模型无需任何目标语言的任务数据，就能直接在其他语言上生成文本，即实现零样本跨语言迁移（zero-shot cross-lingual transfer）。在问题生成和抽象式摘要两项任务上的实验表明，XNLG 超越了基于机器翻译流水线的方法，并能借助高资源语言数据有效提升低资源语言的生成质量。这项工作为跨语言 NLG 研究提供了一个无需依赖多个翻译系统、可端到端训练的新范式。
原文 arXiv:1909.10481;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.10481v3