Net2Net: Accelerating Learning via Knowledge Transfer
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens Google Inc., Mountain View, CA Tianqi Chen is also a PhD student at University of Washington.
Abstract
We introduce techniques for rapidly transferring the information stored in one neural net into another neural net. The main purpose is to accelerate the training of a significantly larger neural net. During real-world workflows, one often trains very many different neural networks during the experimentation and design process. This is a wasteful process in which each new model is trained from scratch. Our Net2Net technique accelerates the experimentation process by instantaneously transferring the knowledge from a previous network to each new deeper or wider network. Our techniques are based on the concept of function-preserving transformations between neural network specifications. This differs from previous approaches to pre-training that altered the function represented by a neural net when adding layers to it. Using our knowledge transfer mechanism to add depth to Inception modules, we demonstrate a new state of the art accuracy rating on the ImageNet dataset.
中文速览
在大型神经网络的实际开发中，每次调整模型结构都要从头训练，既耗时又浪费算力，而 Net2Net 技术正是为了打破这一瓶颈而提出的。它的核心思路是"保函数变换"（function-preserving transformation）：把一个训练好的小网络的参数直接映射到更大的网络中，使得新网络在初始化时就能精确复现原网络的输出，相当于把"知识"瞬间迁移过去，之后再继续训练。具体包含两种操作——Net2WiderNet（把某一层拓宽，即增加更多神经元/卷积通道）和 Net2DeeperNet（插入一个初始化为恒等映射的新层，使网络变深），二者均严格保证变换前后网络函数不变。论文将这一机制应用于 Inception 模块的加深，在 ImageNet 数据集上取得了当时的最优准确率，同时显著缩短了训练时间。这项工作的价值在于：它让"先训小模型、再无缝扩容"的迭代开发流程变得切实可行，也为未来终身学习系统中动态扩展模型容量提供了理论基础。
原文 arXiv:1511.05641;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.05641v4