DyLoRA: Parameter-Efficient Tuning of Pretrained Models using Dynamic Search-Free Low Rank Adaptation
Mojtaba Valipour1,2、Mehdi Rezagholizadeh2、Ivan Kobyzev2、Ali Ghodsi1 \AND{mojtaba.valipour, {mehdi.rezagholizadeh, \AND1: University of Waterloo, 2: Huawei Noah’s Ark Lab
Abstract
With the ever-growing size of pretrained models (PMs), fine-tuning them has become more expensive and resource-hungry. As a remedy, low-rank adapters (LoRA) keep the main pretrained weights of the model frozen and just introduce some learnable truncated SVD modules (so-called LoRA blocks) to the model. While LoRA blocks are parameter-efficient, they suffer from two major problems: first, the size of these blocks is fixed and cannot be modified after training (for example, if we need to change the rank of LoRA blocks, then we need to re-train them from scratch); second, optimizing their rank requires an exhaustive search and effort. In this work, we introduce a dynamic low-rank adaptation (DyLoRA) technique to address these two problems together. Our DyLoRA method trains LoRA blocks for a range of ranks instead of a single rank by sorting the representation learned by the adapter module at different ranks during training. We evaluate our solution on different natural language understanding (GLUE benchmark) and language generation tasks (E2E, DART and WebNLG) using different pretrained models such as RoBERTa and GPT with different sizes. Our results show that we can train dynamic sea
中文速览
预训练大模型的参数规模越来越大，直接微调代价极高，LoRA（低秩适配）虽然只训练少量参数、效率较高，却存在两个顽固痛点：秩（rank）的大小必须在训练前确定，一旦定好就无法修改，想换个秩就得从头重训；而且找到最优秩本身就需要反复试验，耗时耗力。针对这两个问题，作者提出了 DyLoRA（动态低秩适配），核心思路是在训练时借鉴"嵌套丢弃"（nested dropout）的排序思想，让适配模块在一个秩的范围内同时学习，并把最重要的信息排列在低秩位置，从而一次训练就能覆盖多个秩值。在 RoBERTa 和 GPT 等模型上，跨 GLUE、E2E、DART、WebNLG 等多项任务的实验表明，DyLoRA 在性能基本不损失的前提下，比 LoRA 快 4 到 7 倍，且在更宽泛的秩范围内都能稳定表现良好。这项工作让大模型的参数高效微调摆脱了秩搜索的负担，大幅降低了实际部署成本。
原文 arXiv:2210.07558;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.07558v2