Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning
Vishakh Padmakumar 1 Leonard Lausen2 Miguel Ballesteros3 Sheng Zha2 He He12 George Karypis2 1New York University, 2AWS AI, 3AWS AI Labs {lausen, ballemig, zhasheng, hehea, Work done during summer internship at AWS AI.
Abstract
Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost.
中文速览
大规模多任务预训练能让模型在各种下游任务上普遍受益，但研究者们同时也发现，中间任务的选择对下游效果影响极大——这两条结论看似矛盾，背后究竟是任务数量在起作用，还是任务相关性在起作用？作者用29个预微调任务和12个目标任务，按任务格式（序列标注、抽取式问答、分类）分成三组，系统地对比了所有任务组合的两阶段训练效果。结果发现：平均而言，任务数量越多，下游表现确实越好；但如果提前知道目标任务属于哪类，只用同类相关任务预微调，就能以更低的计算代价达到与全量大规模多任务训练相当的效果，而用不相关任务预微调则会让性能大幅下滑，最差情况比直接微调还低20%。这项工作厘清了"规模"与"相关性"各自的贡献，为实际应用中如何高效选择中间任务提供了清晰的实证依据。
原文 arXiv:2204.11117;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.11117v2