Q8BERT: Quantized 8Bit BERT
Ofir ZafrirGuy BoudoukhPeter IzsakMoshe Wasserblat Intel AI Lab {ofir.zafrir, guy.boudoukh, peter.izsak,
Abstract
Recently, pre-trained Transformer [14] based language models such as BERT [3] and GPT [9], have shown great improvement in many Natural Language Processing (NLP) tasks. However, these models contain a large amount of parameters. The emergence of even larger and more accurate models such as GPT2 [8] and Megatron111https://github.com/NVIDIA/Megatron-LM, suggest a trend of large pre-trained Transformer models. However, using these large models in production environments is a complex task requiring a large amount of compute, memory and power resources. In this work we show how to perform quantization-aware training during the fine-tuning phase of BERT in order to compress BERT by $4\times$ with minimal accuracy loss. Furthermore, the produced quantized model can accelerate inference speed if it is optimized for 8bit Integer supporting hardware.
中文速览
大型预训练语言模型 BERT 虽然在各类自然语言处理任务上表现出色，但动辄上亿的参数量让它在实际生产环境中面临内存大、计算慢、功耗高的部署难题。为此，研究者提出在 BERT 微调（fine-tuning）阶段同步引入量化感知训练（quantization-aware training），通过"假量化"模拟推理时的舍入误差，让模型在训练中主动适应 8 位整数（Int8）精度，而非训练完成后再做事后量化。实验在 GLUE 基准的八个任务及 SQuAD 问答任务上均表明，该方法能将模型体积压缩 4 倍，同时保持与原始 32 位浮点模型相差不超过 1% 的精度，远优于直接做事后动态量化的效果。这项工作意义在于为 BERT 乃至其他大型 Transformer 模型的低成本落地提供了一条实用路径——压缩后的模型可直接在支持 Int8 运算的硬件上加速推理，有望让高质量 NLP 服务在边缘设备到数据中心的广泛场景中都能低延迟、低功耗地运行。
原文 arXiv:1910.06188;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.06188v2