Learning to Encode Position for Transformer with Continuous Dynamical Model
Xuanqing Liu†, Hsiang-Fu Yu‡, Inderjit Dhillon§‡, Cho-Jui Hsieh† † UCLA §§\S UT Austin ‡‡{\ddagger} Amazon Inc.
Abstract
We introduce a new way of learning to encode position information for non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which contain inductive bias by loading the input tokens sequentially, non-recurrent models are less sensitive to position. The main reason is that position information among input units is not inherently encoded, i.e., the models are permutation equivalent; this problem justifies why all of the existing models are accompanied by a sinusoidal encoding/embedding layer at the input. However, this solution has clear limitations: the sinusoidal encoding is not flexible enough as it is manually designed and does not contain any learnable parameters, whereas the position embedding restricts the maximum length of input sequences. It is thus desirable to design a new position layer that contains learnable parameters to adjust to different datasets and different architectures. At the same time, we would also like the encodings to extrapolate in accordance with the variable length of inputs. In our proposed solution, we borrow from the recent Neural ODE approach, which may be viewed as a versatile continuous version of a ResNet. This model is capable of
中文速览
传统Transformer模型无法感知词语顺序，必须额外加入位置编码，但现有方案要么是固定的正弦函数（不可学习）、要么是可学习的嵌入矩阵（受最大长度限制），两者难以兼顾灵活性与泛化能力。研究者提出了FLOATER，核心思路是把位置编码建模为一个连续动力系统（Neural ODE），用一个轻量级神经网络来描述位置向量随序列索引演化的规律，只需学习这套"系统动力学"而非为每个位置单独学一个向量。这样一来，FLOATER既是数据驱动、可端到端训练的，又天然支持超出训练长度的序列（具有归纳外推能力），参数量也极为节省，还能无缝插入BERT、RoBERTa等已有预训练模型作为热启动微调。在机器翻译、语言理解和问答等多项NLP任务上，FLOATER均稳定超越基线模型，为非循环序列模型提供了一种更优雅的位置建模范式。
原文 arXiv:2003.09229;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2003.09229v1