TAPAS: Train-less Accuracy Predictor for Architecture Search
R. Istrate IBM Research – Zurich, Switzerland Queen’s University of Belfast, United Kingdom F. Scheidegger IBM Research – Zurich, Switzerland G. Mariani IBM Research – Zurich, Switzerland D. Nikolopoulos Queen’s University of Belfast, United Kingdom C. Bekas IBM Research – Zurich, Switzerland A. C. I. Malossi IBM Research – Zurich, Switzerland
Abstract
In recent years an increasing number of researchers and practitioners have been suggesting algorithms for large-scale neural network architecture search: genetic algorithms, reinforcement learning, learning curve extrapolation, and accuracy predictors. None of them, however, demonstrated high-performance without training new experiments in the presence of unseen datasets. We propose a new deep neural network accuracy predictor, that estimates in fractions of a second classification performance for unseen input datasets, without training. In contrast to previously proposed approaches, our prediction is not only calibrated on the topological network information, but also on the characterization of the dataset-difficulty which allows us to re-tune the prediction without any training. Our predictor achieves a performance which exceeds 100 networks per second on a single GPU, thus creating the opportunity to perform large-scale architecture search within a few minutes. We present results of two searches performed in 400 seconds on a single GPU. Our best discovered networks reach 93.67% accuracy for CIFAR-10 and 81.01% for CIFAR-100, verified by training. These networks are performance c
中文速览
神经网络架构搜索（Neural Architecture Search, NAS）一直面临一个核心痛点：要找到高性能的网络结构，往往需要训练成百上千个候选网络，耗费大量GPU时间和计算资源。这篇论文提出了一个叫做TAPAS的免训练精度预测框架，它通过两个关键设计来绕开这一瓶颈：一是用一个轻量级"探针网络"给输入数据集打一个难度分（DCN），二是维护一个跨数据集的历史实验数据库，在遇到新数据集时，根据难度分筛选相似数据集的历史经验来校准预测，整个预测过程无需对新网络做任何训练。实验表明，TAPAS在单块GPU上每秒可评估100个以上的网络结构，400秒内完成大规模架构搜索，最终发现的网络在CIFAR-10和CIFAR-100上分别达到93.67%和81.01%的验证精度，与业界顶尖方法持平。这项工作的意义在于，它将原本需要数百块GPU运行数天的架构搜索压缩到了几分钟，让NAS真正具备了面向普通研究者的实用性。
原文 arXiv:1806.00250;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.00250v1