Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs
Sachin Kumar、Yulia Tsvetkov Language Technologies Institute Carnegie Mellon University
Abstract
The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation. However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint. We propose a general technique for replacing the softmax layer with a continuous embedding layer. Our primary innovations are a novel probabilistic loss, and a training and inference procedure in which we generate a probability distribution over pre-trained word embeddings, instead of a multinomial distribution over the vocabulary obtained via softmax. We evaluate this new class of sequence-to-sequence models with continuous outputs on the task of neural machine translation. We show that our models train up to 2.5x faster than the state-of-the-art models while achieving comparable translation quality. These models are capable of handling very large vocabularies without compromising on translation quality or speed. They also produce more meaningful errors than the softmax-based models, as these errors typically lie in a subspace of the vector space of the reference translations111The code is at https://git
中文速览
序列到序列模型在生成文字时几乎都依赖Softmax层，但这一层计算开销巨大、内存占用高，还逼着研究者把词表压缩到几万个高频词，稀有词因此被粗暴替换成"未知词"符号。为此，作者提出用连续词向量（word embedding）替换Softmax输出层：模型每一步直接生成一个低维向量，再在预训练词向量空间中找最近邻来确定输出词，同时设计了一种基于von Mises-Fisher分布的新概率损失函数，从理论上解决了直接用L2损失拟合词向量时效果差的问题。在神经机器翻译任务上，新模型的训练速度比基于Softmax的最优系统最高快2.5倍，翻译质量相当，且能无缝扩展到超大词表而不损失性能。更重要的是，该方法产生的错误在语义上更接近正确答案，为语言生成任务从根本上摆脱Softmax瓶颈提供了一条可行路径。
原文 arXiv:1812.04616;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1812.04616v3