8-Bit Approximations for Parallelism in Deep Learning
Tim Dettmers The Faculty of Informatics Universià della Svizzera italiana Via Giuseppe Buffi 13, CH-6904 Lugano, Switzerland
Abstract
The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.
中文速览
深度学习在大规模数据集上训练时必须跨多块GPU乃至多台服务器并行，而GPU之间通信带宽有限、延迟较高，成为制约加速比的核心瓶颈。为此，研究者提出将训练过程中传输的32位梯度和非线性激活值压缩为8位近似表示，通过设计结合动态指数与二进制平分树（binary bisection tree）的新型8位数据类型，在压缩比达到4倍的同时尽量保留数值精度。实验表明，在MNIST、CIFAR10和ImageNet上，无论是数据并行还是模型并行，8位近似均未导致预测精度下降，同时带来约2倍的数据传输加速；进一步基于实验数据建立的预测模型显示，在96块GPU组成的集群上，8位方案可实现超过50倍的加速，远超32位方案约23倍的上限，并在模型并行场景下达到当时最优水平。这项工作意义在于，它提供了一条低成本、无损精度的通信压缩路径，让超大规模GPU集群上的卷积网络训练变得切实可行。
原文 arXiv:1511.04561;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.04561v4