ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-shot Generalization
Hanwei Xu∗, Yujun Chen∗, Yulun Du∗, Nan Shao, Yanggang Wang, Haiyu Li, Zhilin Yang† Recurrent AI {xuhanwei, chenyujun, duyulun,
Abstract
We propose a multitask pretraining approach ZeroPrompt for zero-shot generalization, focusing on task scaling and zero-shot prompting. While previous models are trained on only a few dozen tasks, we scale to 1,000 tasks for the first time using real-world data. This leads to a crucial discovery that task scaling can be an efficient alternative to model scaling; i.e., the model size has less impact on performance with an extremely large number of tasks. Our results show that on the datasets we consider, task scaling can improve training efficiency by 30 times in FLOPs. Empirically, ZeroPrompt substantially improves both the efficiency and the performance of zero-shot learning across a variety of academic and production datasets.
中文速览
零样本泛化（zero-shot generalization）长期面临一个困境：要让模型在没见过的任务上表现好，往往只能靠堆更大的模型参数，代价极高。ZeroPrompt 提出了另一条路——把多任务预训练的训练任务数量从以往的几十个大幅扩展到1000个，首次在真实生产数据上系统研究"任务规模扩展"（task scaling）的效果。实验发现，当训练任务足够多时，模型大小对性能的影响显著减弱，一个4亿参数的小模型可以达到与120亿参数大模型相当的零样本表现，计算量（FLOPs）节省约30倍；同时，ZeroPrompt的零样本成绩在多个学术和生产数据集上大幅超越同类大模型，甚至接近有监督微调的RoBERTa-large。这一发现意义重大：它表明扩展任务多样性可以作为扩大模型规模的高效替代方案，为低成本构建实用零样本系统提供了新思路。
原文 arXiv:2201.06910;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2201.06910v2