What Regularized Auto-Encoders Learn from the Data Generating Distribution
Guillaume Alain and Yoshua Bengio Department of Computer Science and Operations Research University of Montreal Montreal, H3C 3J7, Quebec, Canada
Abstract
What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alter
中文速览
自动编码器（auto-encoder）到底在学什么？这篇论文从理论上证明：用一种带收缩正则项的重建误差来训练自动编码器时，模型实际上在学习数据密度的"得分函数"（score function），即对数密度关于输入的梯度，而不是像此前一些解读所认为的那样在学能量函数。研究者进一步揭示了去噪自动编码器（denoising auto-encoder，DAE）与收缩自动编码器（contractive auto-encoder，CAE）在小噪声极限下的等价关系，并给出了完全不依赖网络参数化形式的通用理论定理。有了得分函数的估计，就可以构造近似的 Metropolis-Hastings MCMC 采样过程，直接从模型所隐式定义的分布中采样，而无需计算难以处理的配分函数（partition function）；人工数据集上的实验验证了这一采样方案能有效恢复真实数据分布。这项工作在理论上澄清了重建误差类方法与密度估计之间的本质联系，为把自动编码器用作隐式生成模型提供了坚实基础。
原文 arXiv:1211.4246;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1211.4246v5