Your classifier is secretly an energy based model and you should treat it like one
Will Grathwohl University of Toronto、Vector Institute Google Research、Kuan-Chieh Wang、Jörn-Henrik Jacobsen††University of Toronto、Vector Institute、David Duvenaud University of Toronto、Vector Institute、Kevin Swersky、Mohammad Norouzi Google Research {kswersky, Equal Contribtuion
Abstract
We propose to reinterpret a standard discriminative classifier of $p(y|\mathbf{x})$ as an energy based model for the joint distribution $p(\mathbf{x},y)$ . In this setting, the standard class probabilities can be easily computed as well as unnormalized values of $p(\mathbf{x})$ and $p(\mathbf{x}|y)$ . Within this framework, standard discriminative architectures may be used and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, and out-of-distribution detection while also enabling our models to generate samples rivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and present an approach which adds little overhead compared to standard classification training. Our approach is able to achieve performance rivaling the state-of-the-art in both generative and discriminative learning within one hybrid model.
中文速览
普通分类器的 logit 输出其实暗藏着一个生成模型——把这些 logit 重新解读为数据与标签的联合能量函数（Joint Energy-based Model，JEM），就能同时得到 p(x,y)、p(x) 和 p(y|x) 三个分布，而无需改动任何网络结构。训练时将目标拆分为标准交叉熵加上基于随机梯度朗之文动力学（SGLD）采样的生成建模目标，从而在几乎不增加额外开销的情况下同时优化判别和生成两个方向。实验表明，JEM 在 CIFAR-10 等数据集上的图像生成质量可以媲美近期的 GAN 方法，分类精度也接近最优，同时在模型置信度校准、分布外样本检测和对抗样本鲁棒性三个实用指标上均显著优于纯分类器，甚至持平乃至超越专门为这些任务设计的方法。这项工作说明，让分类器"兼职"做生成模型并非鱼和熊掌不可兼得，而是能带来可观的实际收益，为统一判别与生成学习提供了一条简洁可行的新路径。
原文 arXiv:1912.03263;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1912.03263v3