How Many Data Samples is an Additional Instruction Worth?
Ravsehaj Singh Puri Swaroop Mishra††Mihir Parmar Chitta Baral Arizona State University, Tempe, USA {rpuri8, srmishr1, mparmar3, Equal Contribution
Abstract
Recently introduced instruction-paradigm empowers non-expert users to leverage NLP resources by defining a new task in natural language. Instruction-tuned models have significantly outperformed multitask learning models (without instruction); however they are far from state-of-the-art task-specific models. Conventional approaches to improve model performance via creating datasets with large number of task instances or architectural changes in the model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction-augmentation helpful? We augment a subset of tasks in the expanded version of Natural Instructions with additional instructions and find that it significantly improves model performance (up to 35%), especially in the low-data regime. Our results indicate that an additional instruction can be equivalent to $\sim$ 200 data samples on average across tasks.111Code and dataset is available at https://github.com/Ravsehajsinghpuri/Multi-Variant-Instructions
中文速览
用自然语言描述任务来"教"模型做事的指令范式（instruction paradigm）正在兴起，但这类模型的性能与专门训练的有监督模型仍有较大差距，而非专业用户又很难通过收集大量标注数据或改造模型结构来弥补这一差距。研究者提出了一个简单可行的替代方案：让用户为同一任务写出多个语义相同但措辞各异的指令（指令增强，instruction augmentation），并在涵盖426个NLP任务的Natural Instructions数据集上构建了相应的多变体指令数据集。实验表明，多变体指令在任务专项、多任务和跨任务三种学习设置下分别平均提升了17%、11%和11%的性能，在数据极少的低资源场景下提升幅度最高可达35%；更重要的是，平均而言，多写一条指令变体所带来的性能收益相当于额外提供约200条训练样本。这项工作说明，仅靠改写指令这种几乎零成本的方式，就能显著增强模型能力，为没有技术背景的用户提供了一条实用的性能提升路径。
原文 arXiv:2203.09161;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.09161v3