GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
Elias Frantar IST Austria、Saleh Ashkboos ETH Zurich、Torsten Hoefler ETH Zurich、Dan Alistarh IST Austria、NeuralMagic Corresponding author:
Abstract
Generative Pre-trained Transformer models, known as GPT or OPT, set themselves apart through breakthrough performance across complex language modelling tasks, but also by their extremely high computational and storage costs. Specifically, due to their massive size, even inference for large, highly-accurate GPT models may require multiple performant GPUs, which limits the usability of such models. While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models. In this paper, we address this challenge, and propose GPTQ, a new one-shot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first tim
中文速览
超大型语言模型（如GPT-3，拥有1750亿参数）推理时需要占用数百GB显存，往往要多块高端GPU协同运行，极大限制了实用性。GPTQ提出了一种基于近似二阶信息（second-order information）的一次性权重量化（post-training quantization）方法，核心思想是逐层将权重压缩到3或4比特，同时通过巧妙重用Hessian逆矩阵把计算量降低了几个数量级。实验证明，只需约4个GPU小时，GPTQ就能在几乎不损失准确率的前提下完成OPT-175B或BLOOM-176B的量化，并首次实现了在单张NVIDIA A100上运行1750亿参数模型，推理速度相比FP16最高提升4.5倍。这项工作意义重大，它让原本只能在昂贵多卡集群上运行的顶级大模型，有望在更普惠的硬件上部署，为大模型的广泛落地提供了切实可行的压缩方案。
原文 arXiv:2210.17323;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.17323v2