Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Yelysei Bondarenko, Markus Nagel, Tijmen Blankevoort Qualcomm AI Research {ybond, markusn, Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc.
Abstract
Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges – namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme – per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized t
中文速览
Transformer模型（如BERT）虽然在自然语言处理任务上表现出色，但体积庞大、推理延迟高，难以部署在资源受限的设备上，而量化（quantization）是压缩模型的有效手段，但直接对Transformer做标准8位量化会导致明显的精度下降。研究发现，根本原因在于残差连接中存在结构性异常值（structured outliers）——少数固定的嵌入维度会产生极大的激活值，这些异常值与特殊分隔符[SEP]标记高度相关，并驱使注意力头集中关注该标记，使得不同张量之间的动态范围差异悬殊，标准的逐张量定点格式根本无法兼顾精度与范围。针对这一问题，作者提出了三种互补方案：混合精度量化、全新的"逐嵌入组量化"（per-embedding-group quantization）以及量化感知训练，其中逐嵌入组方案无需额外标注数据即可在训练后直接使用。在GLUE基准测试中，这些方法不仅刷新了BERT训练后量化的最优结果，还将权重和嵌入压缩至2–4位，内存占用缩减8倍以上而精度几乎不损失，为Transformer在端侧高效部署提供了切实可行的技术路径。
原文 arXiv:2109.12948;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.12948v1