Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks
Rabeeh Karimi Mahabadi EPFL University, Idiap Research Institute、Sebastian Ruder DeepMind \ANDMostafa Dehghani Google Brain、James Henderson Idiap Research Institute Work done while the author was at Google.
Abstract
State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. However, such modules are trained separately for each task and thus do not enable sharing information across tasks. In this paper, we show that we can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks, which condition on task, adapter position, and layer id in a transformer model. This parameter-efficient multi-task learning framework allows us to achieve the best of both worlds by sharing knowledge across tasks via hypernetworks while enabling the model to adapt to each individual task through task-specific adapters. Experiments on the well-known GLUE benchmark show improved performance in multi-task learning while adding only $0.29\%$ parameters per task. We additionally demonstrate substantial performance improvements in few-shot domain generalization across a variety of tasks. Our code is publicly available in https://github.com/rabeehk/hyperformer.
中文速览
多任务自然语言处理模型在共享知识与避免任务间负迁移之间长期存在矛盾：单独为每个任务训练适配器（adapter）能防止互相干扰，却无法让任务之间互相借力。本文提出 HyperFormer++，用一个所有任务共享的超网络（hypernetwork）来动态生成各任务、各层专属的适配器参数，超网络的输入同时包含任务嵌入、层编号和适配器位置编号，从而在一套紧凑的参数结构中同时实现跨任务知识共享和任务个性化适配。在标准 GLUE 基准上，该方法在多任务学习场景下超越了 T5 基线和独立适配器方案，且每个新任务平均只需增加 0.29% 的参数；在少样本跨领域泛化实验中也取得了显著提升。这项工作为参数高效的多任务微调提供了一条既省资源又能正向迁移的新路径，对需要在有限数据上同时处理多种任务的实际应用场景具有重要意义。
原文 arXiv:2106.04489;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.04489v1