Task-transformation and Instruction Amelioration to Improve Large Language Model Performance
Swaroop Mishra3fragments3{}^{3}\, Daniel Khashabi1fragments1{}^{1}\, Chitta Baral3fragments3{}^{3}\, Yejin Choi1,2fragmentsfragments1,2{}^{1,2}\, Hannaneh Hajishirzi1,2fragments1,2{}^{1,2} 11{}^{1}Allen Institute for AI 22{}^{2}University of Washington 33{}^{3}Arizona State University
Abstract
What kinds of instructional prompts are easier to follow for Language Models (LMs)? We study this question by conducting extensive empirical analysis that shed light on important features of successful instructional prompts. Specifically, we study several classes of reframing techniques for manual reformulation of prompts into more effective ones. Some examples include decomposing a complex task instruction into multiple simpler tasks or itemizing instructions into sequential steps. Our experiments compare the zero-shot and few-shot performance of LMs prompted with reframed instructions on 12 NLP tasks across 6 categories. Compared with original instructions, our reframed instructions lead to significant improvements across LMs with different sizes. For example, the same reframed prompts boost few-shot performance of GPT3-series and GPT2-series by 12.5% and 6.7% respectively averaged over all tasks. Furthermore, reframed instructions reduce the number of examples required to prompt LMs in the few-shot setting. We hope these empirically-driven techniques will pave the way towards more effective future prompting algorithms.
中文速览
大语言模型（Language Model, LM）在执行自然语言任务时，对提示语（prompt）的写法极为敏感，但究竟哪种写法更容易让模型"听懂"，一直缺乏系统性研究。研究者提出了五类"重构"（reframing）技巧，包括把复杂任务分解成多个子任务、将指令拆分为有序步骤、明确说明任务约束、引入低层次模式以及给出专项说明，以此将原始提示改写成更易理解的形式。在涵盖6大类共12项NLP任务上，针对GPT-2和GPT-3系列模型的大规模实验显示，重构后的提示在零样本和少样本场景下平均分别提升约17%和12.5%，且效果在不同规模的模型上保持一致。这一发现意义重大——无需对模型进行昂贵的微调，仅凭人工改写提示语就能媲美甚至超越使用大量标注数据监督训练的效果，为更高效地利用已有大模型提供了切实可行的低成本路径。
原文 arXiv:2109.07830;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.07830v3