mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Linting Xue Noah Constant Adam Roberts Mihir Kale Rami Al-Rfou Aditya Siddhant Aditya Barua Colin Raffel Google Research Equal Contribution. Please direct correspondence to and
Abstract
The recent ‘‘Text-to-Text Transfer Transformer’’ (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering $101$ languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent ‘‘accidental translation’’ in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.111https://goo.gle/mt5-code
中文速览
现有大型预训练语言模型（如T5）几乎只在英文数据上训练，覆盖了全球80%不讲英语的人口需求极为有限，为此研究团队推出了多语言版本mT5。他们构建了一个基于Common Crawl网页抓取的多语言数据集mC4，涵盖101种语言，并在此基础上按照T5的设计思路预训练了从3亿到130亿参数不等的五档mT5模型。在XNLI、XQuAD、TyDi QA等多项主流多语言基准测试中，mT5均达到了当时的最优水平；研究还发现生成式多语言模型在零样本（zero-shot）场景下容易出现"意外翻译"——模型会把预测结果偷偷翻成错误语言，并提出了在微调时混入无标注预训练数据这一简单方法来有效缓解该问题。mT5的代码与模型权重全部公开，为多语言NLP社区提供了一个强大且通用的基础工具。
原文 arXiv:2010.11934;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.11934v3