On orthogonality and learning recurrent networks with long term dependencies
Eugene Vorontsov Chiheb Trabelsi Samuel Kadoury Chris Pal
Abstract
It is well known that it is challenging to train deep neural networks and recurrent neural networks for tasks that exhibit long term dependencies. The vanishing or exploding gradient problem is a well known issue associated with these challenges. One approach to addressing vanishing and exploding gradients is to use either soft or hard constraints on weight matrices so as to encourage or enforce orthogonality. Orthogonal matrices preserve gradient norm during backpropagation and may therefore be a desirable property. This paper explores issues with optimization convergence, speed and gradient stability when encouraging or enforcing orthogonality. To perform this analysis, we propose a weight matrix factorization and parameterization strategy through which we can bound matrix norms and therein control the degree of expansivity induced during backpropagation. We find that hard constraints on orthogonality can negatively affect the speed of convergence and model performance.
中文速览
深度神经网络和循环神经网络在处理长程依赖任务时常常遭遇梯度消失或爆炸问题，一个流行的应对思路是将权重矩阵约束为正交矩阵（orthogonal matrix），从而让梯度范数在反向传播中保持稳定。这篇论文提出了一种基于奇异值分解（SVD）的权重矩阵分解与参数化方法，通过对奇异值设置可调节的边界（margin），既能在 Stiefel 流形上用测地线梯度下降保持基矩阵的正交性，也能让矩阵有一定自由度偏离严格正交，从而在"完全正交"与"无约束"之间灵活控制。实验在序列复制、数字相加、顺序/乱序 MNIST 分类以及 Penn Treebank 语言建模等任务上系统比较了不同约束强度的效果，结果表明：正交初始化确实有帮助，但把矩阵硬性锁死在正交约束上反而会拖慢收敛速度、损害模型性能，适当放宽约束、允许奇异值在1附近小幅浮动，往往能取得更好的结果。这一发现为如何在梯度稳定性与模型表达能力之间取得平衡提供了新的实验依据，对设计循环神经网络的优化策略具有实践指导意义。
原文 arXiv:1702.00071;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1702.00071v4