Effective Estimation of Deep Generative Language Models
Tom Pelsmaeker ILCC University of Edinburgh、Wilker Aziz ILLC University of Amsterdam Work done while the first author was at the University of Amsterdam. Code is available at https://github.com/tom-pelsmaeker/deep-generative-lm
Abstract
Advances in variational inference enable parameterisation of probabilistic models by deep neural networks. This combines the statistical transparency of the probabilistic modelling framework with the representational power of deep learning. Yet, due to a problem known as posterior collapse, it is difficult to estimate such models in the context of language modelling effectively. We concentrate on one such model, the variational auto-encoder, which we argue is an important building block in hierarchical probabilistic models of language. This paper contributes a sober view of the problem, a survey of techniques to address it, novel techniques, and extensions to the model. To establish a ranking of techniques, we perform a systematic comparison using Bayesian optimisation and find that many techniques perform reasonably similar, given enough resources. Still, a favourite can be named based on convenience. We also make several empirical observations and recommendations of best practices that should help researchers interested in this exciting field.
中文速览
变分自编码器（VAE）用于语言建模时，常常陷入"后验坍塌"（posterior collapse）困境——模型的解码器太强，完全忽略潜变量，退化成普通语言模型，使得潜在语义空间形同虚设。为此，研究者们提出了多种应对策略，包括KL退火、自由比特、InfoVAE、LagVAE等，但这些方法此前从未在统一条件下被系统比较过。本文在全面梳理已有方法的同时，提出了两项新技术：一是基于约束优化的"最小期望速率"（MDR），可在不引入梯度不连续或额外超参数的情况下将互信息控制在预设水平；二是多峰先验，通过先验与近似后验之间的结构性不匹配，从设计上促使模型编码更多信息。通过以贝叶斯优化作为公平的超参数搜索框架进行系统对比，作者发现：在资源充足的情况下，多数方法表现相近，但MDR凭借其简洁性脱颖而出，成为推荐首选——这一结论以及文中总结的最佳实践，为希望在NLP中实际用好VAE的研究者提供了切实可操作的参考。
原文 arXiv:1904.08194;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.08194v3