Mixed Precision Training
Sharan Narang , Gregory Diamos, Erich Elsen Baidu Research {sharan, \ANDPaulius Micikevicius∗, Jonah Alben, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu NVIDIA {pauliusm, alben, dagarcia, bginsburg, mhouston, okuchaiev, gavenkatesh, Equal contributionNow at Google Brain
Abstract
Increasing the size of a neural network typically improves accuracy but also increases the memory and compute requirements for training the model. We introduce methodology for training deep neural networks using half-precision floating point numbers, without losing model accuracy or having to modify hyper-parameters. This nearly halves memory requirements and, on recent GPUs, speeds up arithmetic. Weights, activations, and gradients are stored in IEEE half-precision format. Since this format has a narrower range than single-precision we propose three techniques for preventing the loss of critical information. Firstly, we recommend maintaining a single-precision copy of weights that accumulates the gradients after each optimizer step (this copy is rounded to half-precision for the forward- and back-propagation). Secondly, we propose loss-scaling to preserve gradient values with small magnitudes. Thirdly, we use half-precision arithmetic that accumulates into single-precision outputs, which are converted to half-precision before storing to memory. We demonstrate that the proposed methodology works across a wide variety of tasks and modern large scale (exceeding 100 million parameters
中文速览
用半精度浮点数（FP16）训练深度神经网络，可以降低内存占用、加快计算速度，但FP16的数值范围比单精度（FP32）窄得多，直接使用会导致模型精度严重下降。为此，研究者提出了三项关键技术：始终保留一份FP32的主权重副本用于参数更新、通过损失缩放（loss scaling）防止梯度下溢为零、以及让FP16的矩阵运算在内部以FP32累积中间结果。综合使用这三种方法后，在图像分类、目标检测、语音识别、机器翻译等多项任务上，混合精度训练的最终精度与纯FP32训练完全持平，同时内存占用近乎减半、在支持张量核心的GPU上算术吞吐量最高提升8倍。这项工作为在不改变模型结构和超参数的前提下大幅降低大规模深度学习的训练成本提供了可直接落地的工程方案。
原文 arXiv:1710.03740;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.03740v3