Compositional generalization through meta sequence-to-sequence learning
Brenden M. Lake New York University Facebook AI Reasearch
Abstract
People can learn a new concept and use it compositionally, understanding how to “blicket twice” after learning how to “blicket.” In contrast, powerful sequence-to-sequence (seq2seq) neural networks fail such tests of compositionality, especially when composing new concepts together with existing concepts. In this paper, I show how memory-augmented neural networks can be trained to generalize compositionally through meta seq2seq learning. In this approach, models train on a series of seq2seq problems to acquire the compositional skills needed to solve new seq2seq problems. Meta se2seq learning solves several of the SCAN tests for compositional learning and can learn to apply implicit rules to variables.
中文速览
人类学会一个新词后能立刻把它"组合"进已有语言体系——比如知道"blicket"是什么之后，就能理解"blicket twice"——但现有强大的序列到序列（sequence-to-sequence, seq2seq）神经网络在这类组合泛化测试中往往惨败，尤其是把新概念和旧概念拼合时。为此，作者提出了"元序列到序列学习"（meta seq2seq learning）方法：让记忆增强神经网络在大量小型seq2seq任务上轮番训练，每个训练片段（episode）都提供若干"示范对"存入外部键值记忆，网络据此在不更新权重的情况下直接泛化到新查询序列。实验结果表明，该方法在专门用于测试组合学习的SCAN基准上攻克了多项此前失败的挑战，并能隐式学到作用于变量的规则。这项工作为弥合人类组合认知能力与神经网络之间的鸿沟提供了一条可行路径，对认知科学和自然语言处理均有重要意义。
原文 arXiv:1906.05381;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.05381v2