Learning Factored Representations in a Deep Mixture of Experts
David Eigen 1,2 Marc’Aurelio Ranzato 1 Ilya Sutskever 1 1 Google, Inc. 2 Dept. of Computer Science, Courant Institute, NYU Marc’Aurelio Ranzato currently works at the Facebook AI Group.
Abstract
Mixtures of Experts combine the outputs of several “expert” networks, each of which specializes in a different part of the input space. This is achieved by training a “gating” network that maps each input to a distribution over the experts. Such models show promise for building larger networks that are still cheap to compute at test time, and more parallelizable at training time. In this this work, we extend the Mixture of Experts to a stacked model, the Deep Mixture of Experts, with multiple sets of gating and experts. This exponentially increases the number of effective experts by associating each input with a combination of experts at each layer, yet maintains a modest model size. On a randomly translated version of the MNIST dataset, we find that the Deep Mixture of Experts automatically learns to develop location-dependent (“where”) experts at the first layer, and class-specific (“what”) experts at the second layer. In addition, we see that the different combinations are in use when the model is applied to a dataset of speech monophones. These demonstrate effective use of all expert combinations.
中文速览
让神经网络又大又快一直是个难题——参数多了计算就慢，深度混合专家模型（Deep Mixture of Experts, DMoE）正是为此而来：把多组"专家"子网络按层叠加，每一层都配一个"门控"网络，根据输入动态选择不同专家组合，使模型的有效专家数随层数指数级增长，却只需激活其中一小部分来处理每个样本。为了防止训练初期少数专家垄断所有输入，作者加入了均衡约束，让每个专家被分配到的任务量大致相当，训练稳定后再解除约束做微调。在随机平移版 MNIST 上，模型第一层自动学会了"在哪儿"（位置感知）专家，第二层学会了"是什么"（类别感知）专家，两层形成互补分工，所有专家组合都被充分利用；语音单音素数据集上也观察到类似的分工现象。这项工作验证了通过分层门控可以让大模型在推理时只激活少量参数的可行性，为后续构建更大规模的稀疏条件计算网络奠定了基础。
原文 arXiv:1312.4314;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1312.4314v3