Recurrent Highway Networks
Julian Georg Zilly Rupesh Kumar Srivastava Jan Koutník Jürgen Schmidhuber
Abstract
Many sequential processing tasks require complex nonlinear transition functions from one step to the next. However, recurrent neural networks with "deep" transition functions remain difficult to train, even when using Long Short-Term Memory (LSTM) networks. We introduce a novel theoretical analysis of recurrent networks based on Geršgorin’s circle theorem that illuminates several modeling and optimization issues and improves our understanding of the LSTM cell. Based on this analysis we propose Recurrent Highway Networks, which extend the LSTM architecture to allow step-to-step transition depths larger than one. Several language modeling experiments demonstrate that the proposed architecture results in powerful and efficient models. On the Penn Treebank corpus, solely increasing the transition depth from 1 to 10 improves word-level perplexity from 90.6 to 65.4 using the same number of parameters. On the larger Wikipedia datasets for character prediction (text8 and enwik8), RHNs outperform all previous results and achieve an entropy of 1.27 bits per character.
中文速览
循环神经网络（RNN）在处理序列数据时，每个时间步内部的状态转换通常只有一层非线性变换，深度不足，导致模型表达能力受限；而直接叠加多层非线性转换又会引发严重的梯度消失或爆炸问题，难以训练。作者借助格尔什戈林圆盘定理（Geršgorin circle theorem）对RNN的梯度传播进行了新的数学分析，揭示了LSTM门控机制能够动态调节雅可比矩阵特征值分布、从而缓解梯度问题的本质原因，并在此基础上提出了循环高速公路网络（Recurrent Highway Networks，RHN）——将Highway层嵌入循环状态转换中，使每个时间步内的转换深度可以超过1层，同时保持类似LSTM的可训练性。实验表明，在参数量不变的情况下，仅将转换深度从1增加到10，Penn Treebank数据集上的词级困惑度就从90.6大幅降至65.4，在更大规模的维基百科字符预测任务上也刷新了当时的最优结果。这项工作为构建更强大、更高效的序列模型提供了理论依据和实用方案，证明在时间维度之外增加"空间深度"是提升RNN能力的有效路径。
原文 arXiv:1607.03474;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1607.03474v5