ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
Han Cai, Ligeng Zhu, Song Han Massachusetts Institute of Technology {hancai, ligeng,
Abstract
Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. $10^{4}$ GPU hours) makes it difficult to directly search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a result, they need to utilize proxy tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on the target task. In this paper, we present ProxylessNAS that can directly learn the architectures for large-scale target tasks and target hardware platforms. We address the high memory consumption issue of differentiable NAS and reduce the computational cost (GPU hours and GPU memory) to the same level of regular training while still allowing a large candidate set. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of di
中文速览
传统神经架构搜索（NAS）需要花费海量算力（动辄上万GPU小时）才能找到好的网络结构，为了省钱只能在小数据集或简化任务上做"代理搜索"，再把结果迁移到真实任务，这就难以保证迁移后的架构还是最优的。ProxylessNAS的核心思路是把超参数化的"包含所有候选路径的大网络"通过路径二值化（path binarization）压缩到每次只激活一条路径，使显存消耗降至与正常训练单个网络相当的水平，从而可以直接在ImageNet这样的大规模数据集上搜索，彻底去掉代理任务。此外，论文还把硬件延迟建模为可微的连续函数作为正则化损失，使搜索出的架构能针对GPU、CPU、手机等不同硬件平台各自定制。实验结果表明，在CIFAR-10上仅用5.7M参数就达到2.08%的错误率，优于参数量多6倍的AmoebaNet-B；在ImageNet上比MobileNetV2高出3.1%的Top-1准确率且推理速度快1.2倍——这项工作让"直接在目标任务和目标硬件上搜架构"真正变得可行，对工业落地有重要意义。
原文 arXiv:1812.00332;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1812.00332v2