Sequential Neural Models with Stochastic Layers
Marco Fraccaro† Søren Kaae Sønderby‡ Ulrich Paquet* Ole Winther†‡ † Technical University of Denmark ‡ University of Copenhagen * Google DeepMind
Abstract
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model’s posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the state of the art results on the Blizzard and TIMIT speech modeling data sets by a large margin, while achieving comparable performances to competing methods on polyphonic music modeling.
中文速览
传统循环神经网络（RNN）用确定性隐状态建模序列数据，而状态空间模型（SSM）虽然能对隐状态的不确定性建模，却难以捕捉长程依赖，两者各有短板。这项工作提出了随机循环神经网络（Stochastic Recurrent Neural Network, SRNN），将一个确定性的GRU网络与一个非线性状态空间模型显式地分层叠加：GRU负责记忆长程信息，SSM在其基础上引入随机隐变量来刻画不确定性，两者角色清晰、互不干扰。这种干净的分层结构使得真实后验分布的因式分解规律一目了然，从而可以设计出与之结构匹配的后向递归变分推断网络，在每个时间步同时利用过去和未来的观测信息进行高效的"平滑"推断。在语音建模基准测试（Blizzard 和 TIMIT）上，SRNN 的性能大幅超越此前最优方法，在复调音乐建模上也与顶尖模型持平，证明了在 RNN 中以结构化方式传播不确定性的有效性。
原文 arXiv:1605.07571;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1605.07571v2