Context-Tuning: Learning Contextualized Prompts for Natural Language Generation
Tianyi Tang1,4, Junyi Li1,3, Wayne Xin Zhao1,4,5 🖂 🖂 Corresponding author Ji-Rong Wen1,2,4 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Information, Renmin University of China 3DIRO, Université de Montréal 4Beijing Key Laboratory of Big Data Management and Analysis Methods 5Beijing Academy of Artificial Intelligence, Beijing, 100084, China
Abstract
Recently, pretrained language models (PLMs) have had exceptional success in language generation. To leverage the rich knowledge encoded by PLMs, a simple yet powerful paradigm is to use prompts in the form of either discrete tokens or continuous embeddings. In existing studies, these prompting methods are typically independent of the inputs, lacking sufficient consideration of input semantics. To address this issue, we propose a novel continuous prompting approach, called context-tuning, to fine-tuning PLMs for natural language generation. Firstly, the prompts are derived based on the input text to elicit useful knowledge from PLMs for generation. We refer to such prompts as contextualized prompts. Secondly, we use continuous inverse prompting to improve the process of natural language generation by modeling an inverse generation process from output to input, making the generated text more relevant to the inputs. Furthermore, we utilize a lightweight context-tuning method that fine-tunes only 0.12% of the parameters while maintaining good performance. Our code is publicly available at https://github.com/RUCAIBox/Context-Tuning.
中文速览
预训练语言模型（PLM）在文本生成任务上表现出色，但现有的提示（prompting）方法通常对所有输入使用相同的静态提示，完全忽略了具体输入内容的语义信息，导致生成结果与输入关联性不足。为此，研究者提出了 Context-Tuning 方法：首先，用 BERT 的掩码填充能力，根据具体输入动态生成"情境化提示"（contextualized prompts），让提示包含与当前输入相关的背景知识；其次，引入"连续逆向提示"（continuous inverse prompting），通过对输出到输入的逆向生成过程建模，确保生成文本始终紧扣输入主题；此外，该方法还提供了一个轻量级训练策略，只微调模型 0.12% 的参数便能达到接近全参数微调的效果。在故事生成、评论生成等四项文本生成任务上的实验表明，Context-Tuning 在生成质量和输入相关性上均优于已有基线，这为如何让 PLM 的提示方法真正"读懂"输入、按需调用知识提供了新思路。
原文 arXiv:2201.08670;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2201.08670v2