Efficient Architecture Search by Network Transformation
Han Cai1, Tianyao Chen1, Weinan Zhang1, Yong Yu1, Jun Wang2 1Shanghai Jiao Tong University, 2University College London Correspondence to Weinan Zhang.
Abstract
Techniques for automatically designing deep neural network architectures such as reinforcement learning based approaches have recently shown promising results. However, their success is based on vast computational resources (e.g. hundreds of GPUs), making them difficult to be widely used. A noticeable limitation is that they still design and train each network from scratch during the exploration of the architecture space, which is highly inefficient. In this paper, we propose a new framework toward efficient architecture search by exploring the architecture space based on the current network and reusing its weights. We employ a reinforcement learning agent as the meta-controller, whose action is to grow the network depth or layer width with function-preserving transformations. As such, the previously validated networks can be reused for further exploration, thus saves a large amount of computational cost. We apply our method to explore the architecture space of the plain convolutional neural networks (no skip-connections, branching etc.) on image benchmark datasets (CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method can design highly competitive networks t
中文速览
自动神经网络架构搜索（Neural Architecture Search）虽然已能设计出媲美人工调参的模型，但动辄需要数百块 GPU、且每次都从头训练新网络，代价极为高昂。这篇论文提出了一个名为 EAS（Efficient Architecture Search）的框架，核心思路是：不从零开始设计网络，而是在已有训练好的网络基础上，用强化学习控制器决定"把某层变宽"或"插入新层"，并通过保函数变换（function-preserving transformation）直接复用旧网络的权重来初始化新网络，从而大幅减少重新训练的开销。在仅用 5 块 GPU 的条件下，EAS 在 CIFAR-10 图像分类任务上设计出的纯卷积网络（无跳跃连接）测试错误率达到 4.23%，超越了绝大多数采用复杂结构的现代网络；进一步将该方法应用于 DenseNet 架构空间，还能在参数更少的情况下刷新 DenseNet 的最佳成绩。这项工作的意义在于，它把架构搜索的计算门槛降低到普通研究团队可承受的水平，同时为"在已有模型上持续进化"而非"每次推倒重来"这一更高效的搜索范式提供了有力佐证。
原文 arXiv:1707.04873;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.04873v2