WARP: Word-level Adversarial ReProgramming
Karen Hambardzumyan1, Hrant Khachatrian1,2, Jonathan May3 1YerevaNN, 2Yerevan State University, 3Information Sciences Institute, University of Southern California
Abstract
Transfer learning from pretrained language models recently became the dominant approach for solving many NLP tasks. A common approach to transfer learning for multiple tasks that maximize parameter sharing trains one or more task-specific layers on top of the language model. In this paper, we present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. Adversarial reprogramming attempts to learn task-specific word embeddings that, when concatenated to the input text, instruct the language model to solve the specified task. Using up to 25K trainable parameters per task, this approach outperforms all existing methods with up to 25M trainable parameters on the public leaderboard of the GLUE benchmark. Our method, initialized with task-specific human-readable prompts, also works in a few-shot setting, outperforming GPT-3 on two SuperGLUE tasks with just 32 training samples.
中文速览
预训练语言模型的迁移学习通常需要对数百万参数进行微调，计算和存储成本很高，而本文提出的 WARP（词级对抗重编程，Word-level Adversarial ReProgramming）方法另辟蹊径：只训练插入输入文本中的少量可学习提示词嵌入（prompt embeddings），让冻结的语言模型"照着提示"完成下游任务，每个任务最多只需 25K 个可训练参数。在 GLUE 基准测试的公开排行榜上，WARP 以高达 81.6 的得分超越了所有使用最多 2500 万参数的已有方法，参数量节省了三个数量级；在仅有 32 条训练样本的少样本场景下，WARP 在两个 SuperGLUE 任务上还超越了拥有 1750 亿参数的 GPT-3。这项工作说明，精心设计的输入提示足以充分调动大模型的能力，为多任务低成本部署提供了一条极具实用价值的新路径。
原文 arXiv:2101.00121;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.00121v2