AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
Bo Chang University of British Columbia Vancouver, BC, Canada、Minmin Chen Google Brain Mountain View, CA, USA、Eldad Haber University of British Columbia Vancouver, BC, Canada、Ed H. Chi Google Brain Mountain View, CA, USA Work performed while interning at Google Brain.
Abstract
Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.
中文速览
循环神经网络（RNN）在处理序列数据时面临长程依赖难以学习的痼疾，根源在于反向传播时梯度的指数级爆炸或消失。作者将RNN与常微分方程（ODE）的数值求解联系起来，发现若ODE的雅可比矩阵特征值实部恰好为零（即"临界准则"），系统既稳定又不会耗散信号，从而自然规避梯度问题；反对称矩阵（antisymmetric matrix）的特征值恰好全为纯虚数，完美满足这一条件。基于此，他们将隐藏层权重矩阵构造为反对称形式，通过前向欧拉离散化直接得到一种新架构AntisymmetricRNN，无需额外的门控或正交约束等繁重操作。实验表明，该模型在需要长程记忆的任务上超越标准LSTM，在短程依赖主导的任务上以更少参数持平，且训练动态更稳定可预测。这项工作提供了一条从动力系统稳定性理论出发、以极低额外计算代价解决RNN长程依赖问题的清晰路径。
原文 arXiv:1902.09689;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.09689v1