Muppet: Massive Multi-task Representations with Pre-Finetuning
Armen Aghajanyan Facebook \AndAnchit Gupta Facebook \AndAkshat Shrivastava Facebook \AndXilun Chen Facebook \ANDLuke Zettlemoyer Facebook \AndSonal Gupta Facebook
Abstract
We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
中文速览
大规模语言模型通常经历"预训练→微调"两个阶段，但如何充分利用已有的大量标注数据来进一步提升模型的泛化能力，一直是个难题。这篇论文在两个阶段之间插入了一个新的"预微调（pre-finetuning）"阶段：用横跨约50个数据集、超过480万条标注样本的大规模多任务学习（massively multi-task learning）来精炼语言模型的表示，并专门设计了任务异构批次（task-heterogeneous batches）和损失缩放（loss scaling）等训练技巧来解决多任务训练不稳定的问题。实验表明，无论是判别式模型RoBERTa还是生成式模型BART，经过预微调后在句子预测、常识推理、机器阅读理解、文本摘要等多类任务上均持续涨点，在RTE和HellaSWAG上刷新了当时的最优结果，且在标注数据稀少时提升尤为显著。这项工作揭示了一个关键规律：多任务学习存在"临界点"，任务数量超过约15个后性能才开始随任务数线性提升，为如何真正发挥多任务学习潜力提供了重要参考。
原文 arXiv:2101.11038;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.11038v1