Learning and Evaluating General Linguistic Intelligence
Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, Phil Blunsom DeepMind, London, United Kingdom
Abstract
We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language’s lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on ho
中文速览
当前NLP模型虽然在各项任务上刷出亮眼成绩，却并不真正具备"通用语言智能"——即把已学到的词汇、句法、语义和语用知识快速迁移到新任务的能力。为了系统衡量这一能力，研究者提出了一套基于在线编码（online/prequential coding）的新评估指标，能够量化模型在获得少量新任务样本时的学习速度，而不仅仅看最终性能。通过对BERT、ELMo等主流模型在阅读理解、自然语言推理等多项任务上的大规模实验，研究发现：这些模型依然需要大量领域内标注数据才能表现良好，跨数据集泛化能力很弱（比如在SQuAD上训练的模型换个问答数据集就大幅掉点），并且在持续学习多任务时会出现严重的灾难性遗忘（catastrophic forgetting）。这项工作揭示了当前"刷榜"式评估体系的根本缺陷，并为如何真正构建具有通用语言智能的模型指明了方向。
原文 arXiv:1901.11373;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.11373v1