WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
Sidak Pal Singh ETH Zurich, Switzerland Dan Alistarh IST Austria、Neural Magic, Inc. Work done while at IST Austria.
Abstract
Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian.
中文速览
深度神经网络里的参数剪枝（pruning）一直依赖对损失函数二阶导数（Hessian矩阵）的估计，但现有近似方法（对角、分块、Kronecker分解等）精度究竟几何，学界知之甚少。这篇工作先通过可视化分析发现经验Fisher信息矩阵与Hessian矩阵在结构上高度相似，随后提出WoodFisher方法——利用Woodbury矩阵恒等式对经验Fisher矩阵做高效递推求逆，从而得到既忠实又可扩展的逆Hessian向量积估计。将WoodFisher嵌入经典的Optimal Brain Damage/Surgeon剪枝框架后，在ImageNet上对ResNet-50、MobileNet等主流网络进行测试，无论是单步一次性剪枝还是允许重训练的渐进式剪枝，WoodFisher均超越或持平当前最优方法。这项工作表明，借助对Fisher矩阵结构的精准利用，二阶方法完全可以在工业规模的模型压缩中发挥实际优势，同时还能自动确定逐层稀疏度并在数据极度受限的场景下工作。
原文 arXiv:2004.14340;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.14340v5