Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
KyungHyun Cho Université de Montréal、Yoshua Bengio Université de Montréal CIFAR Fellow
Abstract
Many state-of-the-art results obtained with deep networks are achieved with the largest models that could be trained, and if more computation power was available, we might be able to exploit much larger datasets in order to improve generalization ability. Whereas in learning algorithms such as decision trees the ratio of capacity (e.g., the number of parameters) to computation is very favorable (up to exponentially more parameters than computation), the ratio is essentially 1 for deep neural networks. Conditional computation has been proposed as a way to increase the capacity of a deep neural network without increasing the amount of computation required, by activating some parameters and computation “on-demand”, on a per-example basis. In this note, we propose a novel parametrization of weight matrices in neural networks which has the potential to increase up to exponentially the ratio of the number of parameters to computation. The proposed approach is based on turning on some parameters (weight matrices) when specific bit patterns of hidden unit activations are obtained. In order to better control for the overfitting that might result, we propose a parametrization that is tree-st
中文速览
深度神经网络的性能瓶颈在于：模型参数量与计算量几乎是1:1的关系，想要更强的模型就得花更多的算力。本文提出一种新的神经网络权重矩阵参数化方案，核心思路是根据隐层神经元激活值的二进制模式（即"门控位"）来动态选择不同的权重矩阵，从而让参数量相对计算量呈指数级增长——用k个门控比特就能索引多达2^k套权重，而实际计算量几乎不变。为了防止因参数过多导致过拟合，作者进一步设计了一种树形前缀求和结构，让短比特前缀对应的权重被更多样本更新、长前缀对应的权重只做小幅修正，类似语言模型中的回退平滑策略，并配合懒惰权重衰减实现高效正则化。这一机制让"条件计算"（conditional computation）真正落地，为在不大幅增加推理开销的前提下训练远超当前规模的深度网络提供了一条可行路径。
原文 arXiv:1406.7362;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1406.7362v1