Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
Jiaxiang Wu Weidong Huang Junzhou Huang Tong Zhang
Abstract
Large-scale distributed optimization is of great importance in various applications. For data-parallel based distributed learning, the inter-node gradient communication often becomes the performance bottleneck. In this paper, we propose the error compensated quantized stochastic gradient descent algorithm to improve the training efficiency. Local gradients are quantized to reduce the communication overhead, and accumulated quantization error is utilized to speed up the convergence. Furthermore, we present theoretical analysis on the convergence behaviour, and demonstrate its advantage over competitors. Extensive experiments indicate that our algorithm can compress gradients by a factor of up to two magnitudes without performance degradation.
中文速览
大规模分布式深度学习中，各节点每轮训练都要互相传输梯度，当模型参数量庞大时，这一通信开销会严重拖慢整体速度。作者提出了一种名为 ECQ-SGD（误差补偿量化随机梯度下降，Error Compensated Quantized SGD）的算法：在每次传输前先将梯度量化成低比特表示以压缩数据量，同时把历史所有迭代中积累的量化误差叠加回当前梯度再做量化，从而抵消误差累积对收敛的拖累。理论分析证明，尽管 ECQ-SGD 的量化梯度是有偏估计、方差界看似更大，但通过引入误差反馈机制，其最坏情况下的次优性误差界反而比无误差补偿的 QSGD 更紧。大量实验表明，该算法可将梯度通信量压缩至原来的约百分之一甚至千分之一，而模型精度几乎不受损失，为在带宽受限的集群上高效训练大模型提供了切实可行的方案。
原文 arXiv:1806.08054;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.08054v1