P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
Xiao Liu1,2∗, Kaixuan Ji1∗, Yicheng Fu1∗, Weng Lam Tam1, Zhengxiao Du1,2, Zhilin Yang1,3†, Jie Tang1,2† 1Tsinghua University, KEG 2Beijing Academy of Artificial Intelligence (BAAI) 3Shanghai Qi Zhi Institute
Abstract
Prompt tuning, which only tunes continuous prompts with a frozen language model, substantially reduces per-task storage and memory usage at training. However, in the context of NLU, prior work reveals that prompt tuning does not perform well for normal-sized pretrained models. We also find that existing methods of prompt tuning cannot handle hard sequence labeling tasks, indicating a lack of universality. We present a novel empirical finding that properly optimized prompt tuning can be universally effective across a wide range of model scales and NLU tasks. It matches the performance of finetuning while having only 0.1%-3% tuned parameters. Our method P-Tuning v2 is an implementation of Deep Prompt Tuning Li and Liang (2021); Qin and Eisner (2021) optimized and adapted for NLU. Given the universality and simplicity of P-Tuning v2, we believe it can serve as an alternative to finetuning and a strong baseline for future research.111Our code and data are released at https://github.com/THUDM/P-tuning-v2.
中文速览
提示调优（prompt tuning）本是一种只训练少量连续提示词、冻结大模型参数的高效方法，但此前研究发现它在中小规模模型和序列标注等复杂任务上表现远不如全参数微调（fine-tuning），适用范围有限。本文提出 P-Tuning v2，核心做法是将可训练的连续提示词从仅插入输入层扩展到模型每一个 Transformer 层（即深度提示调优，Deep Prompt Tuning），同时配合提示长度调整、移除 verbalizer 改用线性分类头等优化细节。实验覆盖 330M 到 10B 参数的多种预训练模型，以及 SuperGLUE 分类任务、命名实体识别、抽取式问答、语义角色标注等多类任务，结果表明 P-Tuning v2 在所有规模和任务上均能与全参数微调打平，而可训练参数量仅占微调的 0.1%–3%。这一发现打破了"提示调优只在超大模型上有效"的固有认知，为未来研究提供了一个兼具参数高效性与普适性的强基线方法。
原文 arXiv:2110.07602;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.07602v3