ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
Vamsi Aribandi †, Yi Tay†, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder♣♣\clubsuit, Donald Metzler Google Research, ♣♣\clubsuitDeepMind {aribandi, Google AI Resident. †Equal contribution. Sebastian is now at Google Research. Sanket returned to CMU.
Abstract
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.
中文速览
预训练语言模型通常只依赖自监督目标（如填空、去噪）来学习，但这种方式在常识推理等需要标注信号的能力上进步缓慢。研究者构建了一个名为 ExMix（Extreme Mixture）的超大规模有监督任务集合，涵盖 107 个多样化 NLP 任务、共约 1800 万条标注样本，并将其与自监督预训练目标联合用于预训练，得到了新模型 ExT5。实验发现，直接把所有 107 个任务一起混入预训练，比费力筛选"最优任务子集"效果更好——任务数量越多、批量越大，下游性能就越强，说明负迁移在预训练阶段远不如微调阶段那么致命。最终 ExT5 在 SuperGLUE、GEM、Rainbow、闭卷问答等多个主流基准上超越了强 T5 基线，同时大幅提升了预训练的样本效率，为"用海量有监督任务做预训练"这一新范式提供了有力的实证支撑。
原文 arXiv:2111.10952;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.10952v2