Improving Language Modeling using Densely Connected Recurrent Neural Networks
Fréderic Godin, Joni Dambre and Wesley De Neve IDLab - ELIS, Ghent University - imec, Ghent, Belgium
Abstract
In this paper, we introduce the novel concept of densely connected layers into recurrent neural networks. We evaluate our proposed architecture on the Penn Treebank language modeling task. We show that we can obtain similar perplexity scores with six times fewer parameters compared to a standard stacked 2-layer LSTM model trained with dropout Zaremba et al. (2014). In contrast with the current usage of skip connections, we show that densely connecting only a few stacked layers with skip connections already yields significant perplexity reductions.
中文速览
把"稠密连接"（dense connection）这一图像分类领域的成熟思路引入循环神经网络，是解决多层LSTM堆叠时参数冗余、信息传递受阻问题的关键切入点。研究者将每一层的隐状态直接连接到其后所有层的输入，让高层网络和最终分类层都能直接"看到"词嵌入，从而保留更多底层语义信息。在Penn Treebank语言建模基准上，仅用两层稠密连接LSTM就能将困惑度（perplexity）比同等参数的双层堆叠LSTM降低约9%；而达到与Zaremba等人最佳66M参数模型相当的效果时，稠密连接模型只需11M参数，缩减了六倍。这项工作说明，跳跃连接（skip connection）作为即插即用的结构改进，即使在层数很少时也能显著提升语言模型效率，为构建轻量级序列建模网络提供了简洁有效的新思路。
原文 arXiv:1707.06130;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.06130v1