Multitask Prompted Training Enables Zero-Shot Task Generalization
Victor Sanh Hugging Face \AndAlbert Webson∗ Brown University \AndColin Raffel∗ Hugging Face \AndStephen H. Bach∗ Brown、Snorkel AI \ANDLintang Sutawika BigScience \AndZaid Alyafeai KFUPM \AndAntoine Chaffin IRISA、IMATAG \AndArnaud Stiegler Hyperscience \AndTeven Le Scao Hugging Face \AndArun Raja I2R, Singapore \AndManan Dey SAP \AndM Saiful Bari NTU, Singapore \AndCanwen Xu UCSD、Hugging Face \AndUrmish Thakker SambaNova Systems \AndShanya Sharma Walmart Labs \AndEliza Szczechla BigScience \AndTaewoon Kim VU Amsterdam \AndGunjan Chhablani BigScience \AndNihal V. Nayak Brown University \AndDebajyoti Datta University of Virginia \AndJonathan Chang ASUS \AndMike Tian-Jian Jiang ZEALS, Japan \AndHan Wang NYU \AndMatteo Manica IBM Research \AndSheng Shen UC Berkeley \AndZheng-Xin Yong Brown University \AndHarshit Pandey BigScience \AndMichael McKenna Parity \AndRachel Bawden Inria, France \AndThomas Wang Inria, France \AndTrishala Neeraj BigScience \AndJos Rozen Naver Labs Europe \AndAbheesht Sharma BITS Pilani, India \AndAndrea Santilli University of Rome \AndThibault Fevry BigScience \AndJason Alan Fries Stanford、Snorkel AI \AndRyan Teehan Charles River Analytics \AndTali Bers Brown University \AndStella Biderman Booz Allen、EleutherAI \AndLeo Gao EleutherAI \AndThomas Wolf Hugging Face \AndAlexander M. Rush Hugging Face Equal contribution. Full list of individual contributions detailed in Appendix A. Corresponding authors: and
Abstract
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models’ pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to $16\times$ its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6 $\times$ its size. All trained models are available at https://github.com/bigscie
中文速览
让大语言模型不靠"巨无霸"体量也能泛化到没见过的任务——这正是这项研究想解决的核心问题。研究者构建了一个名为"公开提示池"（Public Pool of Prompts, P3）的大规模提示数据集，把来自62个监督学习数据集的任务全部改写成自然语言问答形式，每个数据集配有多个措辞各异的提示模板，然后用这个混合任务集对T5编码器-解码器模型进行有监督的多任务微调，得到模型T0。在完全未见过的四类任务（自然语言推理、共指消解、句子补全、词义消歧）上评测时，T0在11个基准数据集中的9个上达到或超越了GPT-3的零样本表现，而参数量仅为GPT-3的约十六分之一；在BIG-bench难题榜上也以小博大，击败了体量高达自身6倍的模型。这一结果表明，零样本泛化能力并非大模型的专利，通过显式的大规模多任务提示训练就能高效地激发出来，为以更低计算成本构建通用语言模型提供了切实可行的路径。
原文 arXiv:2110.08207;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.08207v3