Adaptive Input Representations for Neural Language Modeling
Alexei Baevski、Michael Auli Facebook AI Research, Menlo Park, CA, USA
Abstract
We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the wikitext-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the billion word benchmark, we achieve 23.02 perplexity.111Code and pre-trained models available at http://github.com/pytorch/fairseq
中文速览
语言模型需要对数十万个词汇同时计算概率，既耗时又消耗大量参数，而罕见词往往因训练样本不足而过拟合。本文提出"自适应输入表示"（adaptive input representations），将此前只用于输出层的自适应 softmax 思想迁移到输入词嵌入层——根据词频分配不同维度的嵌入，高频词用大维度、低频词用小维度，并通过线性投影统一后送入模型，输入输出层还可共享权重以进一步压缩参数。在基于自注意力（self-attention）的架构上，系统对比了词级、字符 CNN 和字节对编码（BPE）等多种输入输出方案，结果显示自适应词嵌入模型的参数量比字符 CNN 少，训练速度快一倍以上，精度更高。在 WikiText-103 基准上困惑度（perplexity）降至 18.7，比此前最佳结果整整低了 10.5，在十亿词基准上也以 23.02 的困惑度刷新了最优纪录，充分证明了这一方案在大规模语言建模中的实用价值。
原文 arXiv:1809.10853;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.10853v3