AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
Wei Wen Duke University , Feng Yan University of Nevada, Reno , Yiran Chen Duke University and Hai Li Duke University
Abstract
Depth is a key component of Deep Neural Networks (DNNs), however, designing depth is heuristic and requires many human efforts. We propose AutoGrow to automate depth discovery in DNNs: starting from a shallow seed architecture, AutoGrow grows new layers if the growth improves the accuracy; otherwise, stops growing and thus discovers the depth. We propose robust growing and stopping policies to generalize to different network architectures and datasets. Our experiments show that by applying the same policy to different network architectures, AutoGrow can always discover near-optimal depth on various datasets of MNIST, FashionMNIST, SVHN, CIFAR10, CIFAR100 and ImageNet. For example, in terms of accuracy-computation trade-off, AutoGrow discovers a better depth combination in ResNets than human experts. Our AutoGrow is efficient. It discovers depth within similar time of training a single DNN. Our code is available at https://github.com/wenwei202/autogrow.
中文速览
深度神经网络（DNN）的层数设计至今靠人工试错，费时费力且难以推广到不同数据集。AutoGrow 提出了一种自动发现网络深度的方法：从最浅的种子网络出发，每隔固定周期就往各子网络中追加新模块，若准确率持续无法提升则停止该子网络的生长，最终自动确定最优深度。关键发现是，相比过去流行的"网络变形"（Network Morphism）方法，在网络尚未收敛时就以随机初始化快速追加新层，反而能跳出浅网络的局部最优、取得更好效果。在 MNIST、CIFAR、ImageNet 等多个数据集和多种网络结构上，AutoGrow 均能以接近训练单个网络的时间代价，发现优于人工设计深度的结构——例如在 ResNet 上找到比人类专家更好的精度与计算量权衡方案——为深度学习架构搜索提供了一条高效且通用的自动化路径。
原文 arXiv:1906.02909;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.02909v5