LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Tim Dettmersλ、Mike Lewis†、Younes Belkada§∓、Luke Zettlemoyer†λ University of Washingtonλ Facebook AI Research† Hugging Face§ ENS Paris-Saclay∓ Majority of research done as a visiting researcher at Facebook AI Research.
Abstract
Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible
中文速览
大型语言模型（LLM）推理时需要消耗海量显存，175B参数的模型通常需要多块高端GPU才能运行，这严重限制了它的可及性。为解决这一问题，研究者提出了LLM.int8()量化方法：先用向量级量化（vector-wise quantization）为矩阵乘法中每个内积分配独立的缩放常数以提升精度，再针对Transformer在超过6.7B参数规模后涌现出的极少数（约0.1%）超大数值特征维度，单独保留16位精度进行混合精度分解（mixed-precision decomposition），其余99.9%的数值仍以8位计算。实验表明，该方法可将175B参数模型的显存占用减半，同时在语言建模困惑度和零样本任务上均不损失任何性能，首次实现了无性能退化的百亿级Int8量化。这一成果意义重大：它让OPT-175B、BLOOM-176B这类超大模型首次可以在单台配备消费级GPU的服务器上运行，大幅降低了前沿大模型的使用门槛。
原文 arXiv:2208.07339;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2208.07339v2