Spike-and-Slab Sparse Coding for Unsupervised Feature Discovery
Ian J. Goodfellow, Aaron Courville, Yoshua Bengio Département d’informatique et de recherche opérationnelle Université de Montréal Montréal, QC H3T 1J4
Abstract
We consider the problem of using a factor model we call spike-and-slab sparse coding (S3C) to learn features for a classification task. The S3C model resembles both the spike-and-slab RBM and sparse coding. Since exact inference in this model is intractable, we derive a structured variational inference procedure and employ a variational EM training algorithm. Prior work on approximate inference for this model has not prioritized the ability to exploit parallel architectures and scale to enormous problem sizes. We present an inference procedure appropriate for use with GPUs which allows us to dramatically increase both the training set size and the amount of latent factors.
中文速览
图像识别中，如何从无标注数据中自动学到好用的特征，一直是个核心难题。研究者提出了一种叫做"尖峰-平板稀疏编码"（spike-and-slab sparse coding，S3C）的生成模型，它融合了稀疏编码与受限玻尔兹曼机（RBM）各自的优点：通过二值"尖峰"变量控制特征是否激活、连续"平板"变量控制激活幅度，两者解耦，从而既能产生稀疏表示，又避免了稀疏编码中正则化强度与稀疏程度相互纠缠的缺陷。由于该模型的精确推断不可解，作者推导了一套结构化变分推断方案，并专门设计了适合 GPU 并行加速的更新算法，使训练规模大幅提升。在 CIFAR-10 图像分类基准上，S3C 的分类精度超越了标准稀疏编码和 ssRBM，并凭借该方法赢得了 NIPS 2011 迁移学习挑战赛冠军，证明这一框架在有监督和半监督学习场景下都具有实用价值。
原文 arXiv:1201.3382;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1201.3382v2