Deep Learning and the Information Bottleneck Principle
Naftali Tishby1,2 Noga Zaslavsky1 1The Edmond and Lilly Safra Center for Brain Sciences, Hebrew University of Jerusalem, Israel.2 School of Engineering and Computer Science, Hebrew University of Jerusalem, Israel. This study was funded by the Israeli Science Foundation center of excellence, the DARPA MSEE project, the Israeli Ministry of Science and Technology, and the Intel Collaborative Research Institute for Computational Intelligence (ICRI-CI).
Abstract
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network’s simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.
中文速览
深度神经网络（DNN）为何能一层层地抽取出有用特征、又该设计多少层、每层放多少神经元——这些基础问题长期缺乏严格的理论解释。这篇论文把"信息瓶颈"（Information Bottleneck，IB）原理引入 DNN 分析框架，将每一隐藏层看作输入变量到输出标签之间的一段马尔可夫压缩过程，用互信息同时衡量每层"保留了多少关于标签的有用信息"和"压缩掉了多少输入的冗余信息"。基于此，作者推导出 DNN 的有限样本泛化界，并指出网络的最优层数与每层的最优宽度，恰好对应信息曲线上发生"相变分岔"的临界点——正是在这些点上，数据的统计结构迫使表示空间发生拓扑跃迁，才需要新的隐藏层。这一视角把深度学习的架构设计从经验试错提升到有理论依据的信息论优化问题，为理解为何深层网络优于浅层网络、以及如何设定网络复杂度提供了全新的原则性框架。
原文 arXiv:1503.02406;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1503.02406v1