Memory Transformer
Mikhail S. Burtsev Neural Networks and Deep Learning Lab Moscow Institute of Physics and Technology Dolgoprudny, Russia、Yuri Kuratov Neural Networks and Deep Learning Lab Moscow Institute of Physics and Technology Dolgoprudny, Russia、Anton Peganov Neural Networks and Deep Learning Lab Moscow Institute of Physics and Technology Dolgoprudny, Russia、Grigory V. Sapunov Intento, Inc. Berkeley, CA 94704
Abstract
Transformer-based models have achieved state-of-the-art results in many natural language processing tasks. The self-attention architecture allows transformer to combine information from all elements of a sequence into context-aware representations. However, information about the context is stored mostly in the same element-wise representations. This might limit the processing of properties related to the sequence as a whole more difficult. Adding trainable memory to selectively store local as well as global representations of a sequence is a promising direction to improve the Transformer model. Memory-augmented neural networks (MANNs) extend traditional neural architectures with general-purpose memory for representations. MANNs have demonstrated the capability to learn simple algorithms like Copy or Reverse and can be successfully trained via backpropagation on diverse tasks from question answering to language modeling outperforming RNNs and LSTMs of comparable complexity. In this work, we propose and study few extensions of the Transformer baseline (1) by adding memory tokens to store non-local representations, (2) creating memory bottleneck for the global information, (3) control
中文速览
现有Transformer模型虽然通过自注意力机制能为每个词元生成丰富的上下文表示，但全局信息分散存储在各词元向量中，难以被高效提取和利用。为此，研究者提出在Transformer输入序列前添加专用的记忆词元（memory tokens，[mem]），让模型自主学习将局部或全局信息存入这些"外挂记忆槽"，并在此基础上衍生出带独立记忆控制层的MemCtrl和强制信息只能经由记忆瓶颈传递的MemBottleneck三种变体。在WMT-14德英翻译和语言建模任务上，含记忆词元的模型普遍优于基线Transformer，最佳变体（MemCtrl Shared）在翻译任务上取得了最高BLEU分数；消融实验还证明推理时移除记忆词元会导致性能骤降，说明模型确实在利用这些记忆单元存储关键信息。这项工作提供了一种简单、通用、计算友好的方式来增强Transformer处理全局上下文的能力，对改进长序列建模和缓解注意力计算瓶颈都具有参考价值。
原文 arXiv:2006.11527;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.11527v2