Do Deep Nets Really Need to be Deep? *** Draft for NIPS 2014 (not camera ready copy) ***
Lei Jimmy Ba University of Toronto、Rich Caruana Microsoft Research
Abstract
Currently, deep neural networks are the state of the art on problems such as speech recognition and computer vision. In this paper we empirically demonstrate that shallow feed-forward nets can learn the complex functions previously learned by deep nets and achieve accuracies previously only achievable with deep models. Moreover, in some cases the shallow neural nets can learn these deep functions using the same number of parameters as the original deep models. On the TIMIT phoneme recognition and CIFAR-10 image recognition tasks, shallow nets can be trained that perform similarly to complex, well-engineered, deeper convolutional architectures.
中文速览
深度神经网络在语音和图像识别上的优势，究竟来自"深度"本身，还是来自参数量和训练方式？研究者用"模型压缩"（model compression）的思路来回答这个问题：先训练一个高精度的深度网络作为"教师"，再让只有一个隐藏层的浅层网络（shallow net）去模仿教师的输出——关键技巧是让浅层网络直接回归教师在softmax之前的对数概率值（logit），从而保留教师学到的全部细粒度预测信息。在语音识别基准TIMIT和图像识别基准CIFAR-10上，经过这种"模仿训练"的浅层网络能达到与深度卷积网络相当的准确率，有时参数量也相近；而直接用原始标签训练同样结构的浅层网络，性能则明显更差。这说明深度网络学到的函数本身并不"必须"用深层结构才能表达，真正的瓶颈在于现有优化算法难以直接从原始标签中把浅层网络训练到同等水平，这一发现对理解深度学习为何有效具有重要启示意义。
原文 arXiv:1312.6184;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1312.6184v7