An Analysis of Neural Language Modeling at Multiple Scales
Stephen Merity Nitish Shirish Keskar Richard Socher
Abstract
Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.
中文速览
大规模语言建模其实不一定非得靠新奇复杂的模型架构——把现有的 LSTM 和 QRNN 调好、配上自适应 softmax（adaptive softmax）并绑定权重（weight tying），同样能在字符级和词级语言模型任务上达到当时的最优水平。作者将这套经过精心调参的框架分别应用于字符级数据集 Penn Treebank 和 enwik8，以及大规模词级数据集 WikiText-103，全部实验只用一块现代 GPU，耗时从 12 小时到 2 天不等。结果表明，LSTM 在字符级任务上更具优势，而 QRNN 凭借更高的并行效率在词级大数据集上表现出色。这项工作说明，在追求新架构之前，对基础模型做充分的工程优化和超参调整往往被低估，值得重新审视。
原文 arXiv:1803.08240;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.08240v1