Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
Hakan Inan, Khashayar Khosravi Stanford University Stanford, CA, USA、Richard Socher Salesforce Research Palo Alto, CA, USA
Abstract
Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.
中文速览
循环神经网络（RNN）在语言建模中一直沿用"一hot目标分类"框架，每次训练只告诉模型"答案就是这一个词"，完全忽略了词与词之间的语义相似性，也没有充分利用模型内部已经学到的词向量知识，导致参数浪费且学习效率低下。为此，研究者提出了一个新的损失函数框架：在原有交叉熵损失之外，额外加入一项KL散度损失，让模型的预测分布不仅向真实标签靠拢，还向基于词向量相似度构造的"软目标分布"靠拢，从而把词语之间的语义关联融入训练信号。更进一步，通过对这一增强损失的理论分析，研究者发现输入词嵌入矩阵和输出投影矩阵在最优解处实际上张成同一列空间，这直接推导出可以将两个矩阵绑定共享（weight tying），大幅削减模型参数量。在Penn Treebank等标准基准上的实验表明，仅靠词嵌入矩阵共享便能达到当时最优水平，这不仅验证了理论的合理性，也为后续语言模型设计提供了一个简洁而有据可依的改进思路。
原文 arXiv:1611.01462;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.01462v3