Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Sheng Shen111Equal Contribution Zhen Dong∗ Jiayu Ye∗ Linjian Ma222Work done while interning at Wave Computing. Zhewei Yao Amir Gholami Michael W. Mahoney Kurt Keutzer University of California at Berkeley {sheng.s zhendong yejiayu linjian zheweiy amirgh mahoneymw and
Abstract
Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use a Hessian based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most $2.3\%$ performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to $13\times$ compression of the model parameters, and up to $4\times$ compression of the embedding table as well as activations. Among all tasks, we observed the
中文速览
大规模预训练语言模型BERT虽然在各类自然语言处理任务上效果出色，但动辄数百MB的内存占用和较高的推理延迟让它很难部署在手机、智能助手等资源受限的边缘设备上。为此，研究者提出了Q-BERT方法，核心思路是用二阶海森（Hessian）信息来衡量每一层对量化的敏感程度——曲率越大的层越不能压缩，并同时考虑特征值的均值和方差来分配不同精度，从而实现混合精度量化；此外还设计了一种"分组量化"方案，将权重矩阵按组分别设置量化范围，在不大幅增加硬件复杂度的前提下减少精度损失。在情感分类、自然语言推理、命名实体识别和机器阅读理解四项下游任务上，Q-BERT将模型权重压缩至最低2比特，实现最高13倍的参数压缩和4倍的嵌入层及激活值压缩，性能下降不超过2.3%。这项工作首次证明BERT可以被量化到超低比特而仍保持可用的精度，为在边缘端部署大型语言模型提供了切实可行的方案。
原文 arXiv:1909.05840;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.05840v2