Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
Rabeeh Karimi Mahabadi EPFL University, Idiap Research Institute、James Henderson Idiap Research Institute、Sebastian Ruder DeepMind
Abstract
Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods, but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers.
中文速览
针对大规模预训练语言模型（PLM）在迁移到下游任务时参数低效、存储浪费、低资源场景下不稳定等问题，本文提出了一种名为 Compacter 的轻量微调方法。Compacter 的核心思路是在预训练模型的每一层插入小型任务适配模块，这些模块的权重矩阵通过"Kronecker 积之和"来高效构造——具体来说，用全层共享的"慢权重"矩阵与每层独有的秩一"快权重"矩阵的 Kronecker 积叠加，从而将参数量从普通适配器的 O(kd) 大幅压缩到 O(k+d)。在 T5-Base 模型上，Compacter 仅需训练全模型 0.047% 的参数，却在 GLUE 基准上与全量微调持平，在 SuperGLUE 和低资源场景下甚至超越了全量微调，同时优于其他参数高效微调方法。这一工作为在实际部署中以极低的存储和计算代价让同一个底座模型适配多个任务提供了可行方案。
原文 arXiv:2106.04647;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.04647v2