Deep Temporal Sigmoid Belief Networks for Sequence Modeling
Zhe Gan, Chunyuan Li, Ricardo Henao, David Carlson and Lawrence Carin Department of Electrical and Computer Engineering Duke University, Durham, NC 27708 {zhe.gan, chunyuan.li, r.henao, david.carlson,
Abstract
Deep dynamic generative models are developed to learn sequential dependencies in time-series data. The multi-layered model is designed by constructing a hierarchy of temporal sigmoid belief networks (TSBNs), defined as a sequential stack of sigmoid belief networks (SBNs). Each SBN has a contextual hidden state, inherited from the previous SBNs in the sequence, and is used to regulate its hidden bias. Scalable learning and inference algorithms are derived by introducing a recognition model that yields fast sampling from the variational posterior. This recognition model is trained jointly with the generative model, by maximizing its variational lower bound on the log-likelihood. Experimental results on bouncing balls, polyphonic music, motion capture, and text streams show that the proposed approach achieves state-of-the-art predictive performance, and has the capacity to synthesize various sequences.
中文速览
时序数据中隐藏着复杂的非线性动态依赖关系，传统的隐马尔可夫模型（HMM）和线性动态系统（LDS）都难以有效捕捉这类结构。研究者提出了一种深度动态生成模型，其核心是将多个时序Sigmoid信念网络（Temporal Sigmoid Belief Network, TSBN）按层堆叠，每一层的隐变量偏置由前一时刻的隐状态动态调节，从而同时兼顾时间依赖和层次化表示能力。为了解决这类有向图模型中"explaining away"导致的推断困难，研究者引入了识别模型（recognition model）与生成模型联合训练，通过最大化变分下界实现快速近似推断，并借助方差缩减技术使随机梯度估计更加稳定。在弹跳球视频、多声部音乐、人体运动捕捉和文本流等多个基准任务上，该模型的序列预测性能均达到当时最优水平，并能自主合成多样化的序列，为高维时序数据的概率建模提供了一个统一且可扩展的深度生成框架。
原文 arXiv:1509.07087;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1509.07087v1