Learning deep representations by mutual information estimation and maximization
R Devon Hjelm MSR Montreal, MILA, UdeM, IVADO、Alex Fedorov MRN, UNM、Samuel Lavoie-Marchildon MILA, UdeM、Karan Grewal U Toronto、Phil Bachman MSR Montreal、Adam Trischler MSR Montreal、Yoshua Bengio MILA, UdeM, IVADO, CIFAR
Abstract
This work investigates unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and compares favorably with fully-supervised learning on several classification tasks in with some standard architectures. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation learning objectives for specific end-goals.
中文速览
表征学习领域长期面临一个核心难题：如何在没有标签的情况下让神经网络自动学到对下游任务真正有用的特征。这篇论文提出了一种叫做深度信息最大化（Deep InfoMax, DIM）的方法，核心思路是训练一个编码器，最大化输入数据与编码器输出之间的互信息（mutual information）；关键创新在于不只考虑"全局"互信息，而是同时最大化高层表征与输入局部区域（如图像小块）之间的平均互信息，这样编码器就被迫提取各局部共享的结构性信息，而非无用的像素噪声。此外，DIM还通过对抗训练让表征的分布匹配预先指定的先验，从而控制表征的统计特性（如独立性）。实验表明，DIM在多个分类任务上超越了多种流行的无监督学习方法，甚至在某些标准架构下与全监督方法相当，为灵活地针对不同目标定制表征学习提供了新思路。
原文 arXiv:1808.06670;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1808.06670v5