What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
The BigScience Architecture、Scaling Group Thomas Wang1 Adam Roberts2††Daniel Hesslow3 Teven Le Scao1 Hyung Won Chung2 Iz Beltagy4 Julien Launay3,5 Colin Raffel1†† Hugging Face 2Google 3LightOn 4Allen Institute for AI 5LPENS, École Normale Supérieure Equal contribution.Equal supervision. Individual contributions outlined in Appendix A
Abstract
Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, and there has been limited systematic comparison of these factors. In this work, we present a large-scale evaluation of modeling choices and their impact on zero-shot generalization. In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning. We train models with over 5 billion parameters for more than 170 billion tokens, thereby increasing the likelihood that our conclusions will transfer to even larger scales. Our experiments show that causal decoder-only models trained on an autoregressive language modeling objective exhibit the strongest zero-shot generalization after purely unsupervised pretraining. However, models with non-causal visibility o
中文速览
如今大型语言模型在架构（编码器-解码器、因果解码器、非因果解码器）和预训练目标（自回归语言模型、掩码语言模型）上各有不同，但究竟哪种组合最适合零样本泛化（zero-shot generalization），此前缺乏系统性的大规模比较。研究者训练了超过50亿参数、在1700亿token上预训练的多组模型，全面对比六种"架构×目标"组合在有无多任务微调（multitask finetuning）两种设置下的零样本表现。结果发现：仅做无监督预训练时，因果解码器配自回归目标最强；但加入多任务微调后，以掩码语言模型目标预训练的非因果模型反而全面反超。更重要的是，研究还发现两类模型可以相互"适配"——用自回归目标继续训练一个掩码预训练的非因果模型，可将收敛速度提升1.6倍；反过来将因果解码器改造为非因果模型再做掩码预训练，收敛速度更可加速3.3倍——这为高效获得既擅长生成又擅长多任务微调的模型提供了一条实用捷径。
原文 arXiv:2204.05832;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.05832v1