Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Zhilin Yang , Zihang Dai††, Ruslan Salakhutdinov, William W. Cohen School of Computer Science Carnegie Mellon University Equal contribution. Ordering determined by dice rolling.
Abstract
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.111Code is available at https://github.com/zihangdai/mos.
中文速览
语言模型在预测下一个词时，通常用 Softmax 把隐藏向量和词向量的点积转成概率，但这篇工作从矩阵分解的角度指出：这种做法本质上是在做低秩近似，而自然语言的上下文条件概率矩阵（log probability matrix）极有可能是高秩的，维度受限的 Softmax 根本无法充分表达它，作者把这一根本缺陷命名为"Softmax 瓶颈"（Softmax Bottleneck）。为此，他们提出了一种叫做"Softmax 混合"（Mixture of Softmaxes，MoS）的方法：对同一个上下文计算 K 个 Softmax 分布，再加权平均得到最终概率，这样合成矩阵的秩可以远高于单个 Softmax 的上限，同时参数量增加有限、不易过拟合。实验结果表明，MoS 在 Penn Treebank 和 WikiText-2 上将困惑度（perplexity）分别推进到 47.69 和 40.68，在十亿词大规模数据集上也比基线低了逾 5.6 点，刷新了当时的最优水平。这项工作不仅揭示了主流神经语言模型长期被忽视的容量瓶颈，还提供了一个改动极小却效果显著的解决思路，对后续语言模型设计具有重要的理论和实践参考价值。
原文 arXiv:1711.03953;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.03953v4