Single Path One-Shot Neural Architecture Search with Uniform Sampling
Zichao Guo Equal contribution. This work is done when Haoyuan Mu and Zechun Liu are interns at MEGVII Technology.*1*1 Xiangyu Zhang *1*1 Haoyuan Mu 1122 Wen Heng 11 Zechun Liu 1133 Yichen Wei 11 Jian Sun 11
Abstract
We revisit the one-shot Neural Architecture Search (NAS) paradigm and analyze its advantages over existing NAS approaches. Existing one-shot method, however, is hard to train and not yet effective on large scale datasets like ImageNet. This work propose a Single Path One-Shot model to address the challenge in the training. Our central idea is to construct a simplified supernet, where all architectures are single paths so that weight co-adaption problem is alleviated. Training is performed by uniform path sampling. All architectures (and their weights) are trained fully and equally.
中文速览
神经网络架构自动搜索（Neural Architecture Search, NAS）一直面临计算成本高、难以在ImageNet这样的大规模数据集上落地的瓶颈，根源在于超网络（supernet）训练时各子网络的权重相互耦合，导致优化过程复杂且不稳定。本文提出"单路径一次性"（Single Path One-Shot）方法：将超网络中的每条候选架构设计为独立的单一路径，并采用均匀随机采样策略轮流训练各路径，从根本上解耦了不同操作之间的权重，使训练无需敏感超参数即可稳定收敛。搜索阶段则与训练完全解耦，利用进化算法高效地在已训练好的超网络上独立完成，天然支持FLOPs、延迟等硬约束条件。在ImageNet上的大量实验表明，该方法在精度、搜索效率、内存占用和灵活性上均达到当时最优水平，为工业级NAS应用提供了一套简洁而实用的解决方案。
原文 arXiv:1904.00420;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.00420v4