The Turking Test: Can Language Models Understand Instructions?
Avia Efrat Tel-Aviv University、Omer Levy Tel-Aviv University Facebook AI Research
Abstract
Supervised machine learning provides the learner with a set of input-output examples of the target task. Humans, however, can also learn to perform new tasks from instructions in natural language. Can machines learn to understand instructions as well? We present the Turking Test, which examines a model’s ability to follow natural language instructions of varying complexity. These range from simple tasks, like retrieving the $n$ th word of a sentence, to ones that require creativity, such as generating examples for SNLI and SQuAD in place of human intelligence workers (“turkers”). Despite our lenient evaluation methodology, we observe that a large pretrained language model performs poorly across all tasks. Analyzing the model’s error patterns reveals that the model tends to ignore explicit instructions and often generates outputs that cannot be construed as an attempt to solve the task. While it is not yet clear whether instruction understanding can be captured by traditional language models, the sheer expressivity of instruction understanding makes it an appealing alternative to the rising few-shot inference paradigm.
中文速览
让机器像人一样读懂自然语言指令来完成任务，是AI领域的一大挑战。研究者设计了"Turking测试"（Turking Test）这一系列难度递进的指令理解基准，涵盖从"写出句子第n个词"这样的简单任务，到模拟众包标注者为SNLI、SQuAD等数据集生成样本的复杂任务，并用大型预训练语言模型GPT-2进行测试。结果令人失望：即便采用了非常宽松的评估标准，GPT-2在所有任务上表现都极差，例如在"取第n个词"这种小学生秒答的任务上准确率仅有2%。错误分析显示，模型普遍倾向于无视指令、直接续写输入文本或生成与任务无关的内容，说明当前语言模型对自然语言指令的理解能力极为有限。这项工作揭示了指令理解作为一种比少样本（few-shot）提示更具表达力的学习范式所蕴含的巨大潜力，同时也指出了现有模型在这一方向上仍需跨越的显著鸿沟。
原文 arXiv:2010.11982;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.11982v1