AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2seq Model
Saleh Soltan Shankar Ananthakrishnan Jack FitzGerald Rahul Gupta Wael Hamza Haidar Khan Charith Peris Stephen Rawls Andy Rosenbaum Anna Rumshisky Chandana Satya Prakash Mukund Sridhar Fabian Triefenbach Apurv Verma Gokhan Tur Prem Natarajan Amazon Alexa AI Corresponding author:
Abstract
In this work, we demonstrate that multilingual large-scale sequence-to-sequence (seq2seq) models, pre-trained on a mixture of denoising and Causal Language Modeling (CLM) tasks, are more efficient few-shot learners than decoder-only models on various tasks. In particular, we train a 20 billion parameter multilingual seq2seq model called Alexa Teacher Model (AlexaTM 20B) and show that it achieves state-of-the-art (SOTA) performance on 1-shot summarization tasks, outperforming a much larger 540B PaLM decoder model. AlexaTM 20B also achieves SOTA in 1-shot machine translation, especially for low-resource languages, across almost all language pairs supported by the model (Arabic, English, French, German, Hindi, Italian, Japanese, Marathi, Portuguese, Spanish, Tamil, and Telugu) on Flores-101 dataset. We also show in zero-shot setting, AlexaTM 20B outperforms GPT3 (175B) on SuperGLUE and SQuADv2 datasets and provides SOTA performance on multilingual tasks such as XNLI, XCOPA, Paws-X, and XWinograd. Overall, our results present a compelling case for seq2seq models as a powerful alternative to decoder-only models for Large-scale Language Model (LLM) training.
中文速览
多语言大模型长期被"只有解码器"的架构垄断，但编解码器（seq2seq）结构在处理长上下文时理论上更有优势——亚马逊这项工作正是要验证这一点。研究团队训练了一个200亿参数的多语言seq2seq模型AlexaTM 20B，创新性地将去噪预训练与因果语言建模（Causal Language Modeling）混合，使其同时具备双向理解能力和上下文学习能力，并在12种语言的维基百科与mC4数据上积累了约1万亿词元的训练量。实验结果显示，AlexaTM 20B在1-shot摘要任务上超越了体量大27倍的PaLM 540B，在低资源语言机器翻译（如泰米尔语、泰卢固语）上也以仅1个示例就远超已有监督模型，同时在零样本SuperGLUE、SQuADv2及多项多语言基准上以少8倍的参数量胜过GPT-3 175B。这项工作有力地说明，seq2seq架构并非大模型研究的"次优选项"，在样本效率和长文本处理上甚至更具竞争力，为未来大语言模型的训练范式提供了重要参考。
原文 arXiv:2208.01448;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2208.01448v2