Learn from Instructions instead of Learning just from Data Samples
Swaroop Mishra3 Daniel Khashabi1 Chitta Baral3 Hannaneh Hajishirzi1,2 1Allen Institute for AI 2University of Washington 3Arizona State University Work done while interning at Allen Institute for AI.
Abstract
Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce Natural Instructions, a dataset of 61 distinct tasks, their human-authored instructions, and 193 $k$ task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalizat
中文速览
如何让AI像人类一样仅凭自然语言说明书就能学会从未见过的新任务，是人工智能领域的长期难题。研究者为此构建了"Natural Instructions"数据集，从现有NLP众包标注项目中收集了61种任务的原始众包指令，统一整理成包含任务定义、注意事项、正负例等字段的标准格式，共形成约19.3万条输入输出实例。他们以此为元数据集（meta-dataset），训练BART等生成式预训练语言模型在"见过的任务"上学习如何理解指令，再测试模型能否仅凭指令文本就完成"从未见过的新任务"，结果显示加入指令的模型在跨任务泛化上比不加指令的基线高出约19%，且训练任务种类越多，泛化能力越强。尽管如此，模型表现与人类水平估算上限之间仍存在显著差距，说明"依据自然语言指令学习新任务"这一方向还有巨大提升空间，值得社区持续投入。
原文 arXiv:2104.08773;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.08773v4