AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
Alexandra Peste IST Austria、Eugenia Iofinova IST Austria、Adrian Vladu CNRS、IRIF、Dan Alistarh IST Austria、Neural Magic Correspondence to: Alexandra Peste
Abstract
The increasing computational requirements of deep neural networks (DNNs) have led to significant interest in obtaining DNN models that are sparse, yet accurate. Recent work has investigated the even harder case of sparse training, where the DNN weights are, for as much as possible, already sparse to reduce computational costs during training. Existing sparse training methods are often empirical and can have lower accuracy relative to the dense baseline. In this paper, we present a general approach called Alternating Compressed/DeCompressed (AC/DC) training of DNNs, demonstrate convergence for a variant of the algorithm, and show that AC/DC outperforms existing sparse training methods in accuracy at similar computational budgets; at high sparsity levels, AC/DC even outperforms existing methods that rely on accurate pre-trained dense models. An important property of AC/DC is that it allows co-training of dense and sparse models, yielding accurate sparse–dense model pairs at the end of the training process. This is useful in practice, where compressed variants may be desirable for deployment in resource-constrained settings without re-doing the entire training flow, and also provides
中文速览
训练深度神经网络（DNN）成本高昂，让模型在训练过程中就保持稀疏（sparse training）能节省算力，但现有方法往往精度不达标、缺乏理论保证。这篇论文提出了一种叫做"交替压缩/解压缩训练"（AC/DC，Alternating Compressed/DeCompressed）的方法：训练过程在稠密（dense）阶段和稀疏（sparse）阶段之间循环切换，稀疏阶段只在固定的非零权重子集上做前向和反向传播以省算力，稠密阶段则用于重新调整稀疏结构，两者交替进行直到最终收敛于一个稀疏模型。实验表明，在相同训练轮数下，AC/DC在精度上全面超越现有稀疏训练方法，在高稀疏度（如95%以上）时甚至比依赖预训练稠密模型的后训练剪枝方法更好，同时将训练计算量压缩到稠密基线的45–65%左右。更难得的是，AC/DC能够同时输出一个精确的稀疏模型和一个稠密模型，后者经短暂微调即可恢复原始稠密精度，这让同一套训练流程既能服务于资源受限的部署场景，又为理解稠密与稀疏模型之间的精度差异提供了新视角。
原文 arXiv:2106.12379;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.12379v2