Autoencoding Variational Inference For Topic Models
Akash Srivastava Informatics Forum, University of Edinburgh 10, Crichton St Edinburgh, EH89AB, UK、Charles Sutton Informatics Forum, University of Edinburgh 10, Crichton St Edinburgh, EH89AB, UK Additional affiliation: Alan Turing Institute, British Library, 96 Euston Road, London NW1 2DB
Abstract
Topic models are one of the most popular methods for learning representations of text, but a major challenge is that any change to the topic model requires mathematically deriving a new inference algorithm. A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven difficult to apply to topic models in practice. We present what is to our knowledge the first effective AEVB based inference method for latent Dirichlet allocation (LDA), which we call Autoencoded Variational Inference For Topic Model (AVITM). This model tackles the problems caused for AEVB by the Dirichlet prior and by component collapsing. We find that AVITM matches traditional methods in accuracy with much better inference time. Indeed, because of the inference network, we find that it is unnecessary to pay the computational cost of running variational optimization on test data. Because AVITM is black box, it is readily applied to new topic models. As a dramatic illustration of this, we present a new topic model called ProdLDA, that replaces the mixture model in LDA with a product of experts. By changing only one line of code from LDA, we find that ProdLDA yields much more
中文速览
文本主题模型是分析大规模文本最常用的工具之一，但每当研究者想改动模型假设，就必须重新推导繁琐的数学公式，极大限制了模型创新的速度。为此，作者提出了一种名为 AVITM（Autoencoded Variational Inference for Topic Models）的黑盒推断框架，核心思路是用神经网络（推断网络）直接把文档映射到后验分布，并通过拉普拉斯近似解决 Dirichlet 先验难以重参数化的问题，同时引入批归一化等技巧克服"主题坍塌"的训练陷阱。实验表明，AVITM 在主题质量上与传统均场变分推断和折叠 Gibbs 采样持平，而在测试时仅需一次前向传播，速度大幅领先；基于同一框架，作者只改一行代码便实现了新模型 ProdLDA——将 LDA 中的混合模型替换为专家乘积（product of experts），结果在主题连贯性上显著优于 LDA。这项工作的意义在于：它使研究者无需重推公式就能快速实验各种主题模型变体，为主题建模领域提供了一个像均场和 Gibbs 采样一样可靠的通用推断基础设施。
原文 arXiv:1703.01488;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.01488v1