FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks
Raphael Tang, Ashutosh Adhikari, Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo {r33tang, ashutosh.adhikari,
Abstract
There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, few methods target a specific number of floating-point operations (FLOPs) as part of the optimization objective, despite many reporting FLOPs as part of the results. Furthermore, a one-size-fits-all approach ignores realistic system constraints, which differ significantly between, say, a GPU and a mobile phone—FLOPs on the former incur less latency than on the latter; thus, it is important for practitioners to be able to specify a target number of FLOPs during model compression. In this work, we extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective and show that, given a desired FLOPs requirement, different neural networks can be successfully trained for image classification.
中文速览
神经网络在实际部署中往往因计算量过大而难以落地，现有的模型压缩方法虽然通常会汇报浮点运算次数（FLOPs），却很少将其直接纳入优化目标，导致从业者无法在训练时精确指定目标计算量。本文在一种基于 L0 正则化稀疏化的先进方法基础上，将 FLOPs 约束直接嵌入训练目标，利用硬混凝土分布（hard concrete distribution）处理可微部分，并用得分函数估计器（REINFORCE）处理不可微的 FLOPs 项，从而在单次训练中就能让模型收敛到指定计算量。在 MNIST、CIFAR-10 和 CIFAR-100 上的实验表明，通过调节目标 FLOPs 这一"旋钮"，可以训练出计算量可控的压缩模型，其中在 CIFAR-10 上甚至同时降低了错误率和推理时 FLOPs，达到帕累托改进。这项工作的意义在于让工程师在部署模型时可以直接以计算预算为约束来驱动压缩，而不是事后再去筛选满足要求的结果。
原文 arXiv:1811.03060;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.03060v2