Improving Deep Neural Networks with Probabilistic Maxout Units
Jost Tobias Springenberg and Martin Riedmiller Department of Computer Science University of Freiburg 79110, Freiburg im Breisgau, Germany
Abstract
We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation over a group of linear transformations and is thus partially invariant to changes in its input. Starting from this observation we ask the question: Can the desirable properties of maxout units be preserved while improving their invariance properties ? We argue that our probabilistic maxout (probout) units successfully achieve this balance. We quantitatively verify this claim and report classification performance matching or exceeding the current state of the art on three challenging image classification benchmarks (CIFAR-10, CIFAR-100 and SVHN).
中文速览
针对深度神经网络容易过拟合、激活单元对输入变化不够鲁棒的问题，研究者在经典maxout单元的基础上提出了一种概率化改进版本——probabilistic maxout（probout）单元。核心思路是把maxout中"取k个线性变换里的最大值"这一确定性操作，替换为基于Boltzmann分布的随机采样：每个线性子单元按照softmax概率被选中，而不是每次都强制选最大值，从而让梯度能更均匀地流经所有k个线性子单元，并增强了单元对输入变化的不变性。实验在CIFAR-10、CIFAR-100和SVHN三个图像分类基准上进行，probout在与dropout结合使用时取得了与当时最优方法持平甚至更好的分类准确率。这项工作的意义在于，它在几乎不增加计算开销的前提下，通过一个简单的随机化改造就提升了网络的泛化能力和特征利用率，为设计更鲁棒的激活单元提供了新思路。
原文 arXiv:1312.6116;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1312.6116v2