Techniques for Learning Binary Stochastic Feedforward Neural Networks
Tapani Raiko、Mathias Berglund Department of Information and Computer Science Aalto University Espoo, Finland、Guillaume Alain、Laurent Dinh Department of Computer Science and Operations Research Université de Montréal Montréal, Canada
Abstract
Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output is important. (3) Stochasticity has been shown to be an excellent regularizer, which makes generalization performance potentially better in general. However, training stochastic networks is considerably more difficult. We study training using $M$ samples of hidden activations per input. We show that the case $M=1$ leads to a fundamentally different behavior where the network tries to avoid stochasticity. We propose two new estimators for the training gradient and propose benchmark tests for comparing training algorithms. Our experiments confirm that training stochastic networks is difficult and show that the proposed two estimators perform favorably among all the five known estimators.
中文速览
多层感知机（MLP）中的随机二值隐单元（stochastic binary hidden units）理论上能让网络学习一对多的映射、建模复杂输出结构并起到正则化作用，但如何有效训练这类网络一直是难题。研究者发现，当每个输入只采一个隐层激活样本（M=1）时，网络会主动"逃避"随机性、退化成确定性网络，并从理论上证明了这一现象；只有同时使用多个样本，随机网络才能真正发挥优势。在此基础上，他们提出了两种新的梯度估计方法——一种是偏差更低的近似反向传播，另一种是方差更小的无偏变分估计——并在MNIST手写数字补全和人脸表情预测两个基准任务上与已有的五种估计方法进行了系统比较。实验结果表明训练随机网络确实困难，但两种新估计方法在综合表现上优于现有方法，为构建能捕捉复杂多模态输出分布的神经网络提供了更可靠的训练手段。
原文 arXiv:1406.2989;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1406.2989v3