How to scale distributed deep learning?
Peter H. Jin, Qiaochu Yuan, Forrest Iandola, and Kurt Keutzer Department of Electrical Engineering and Computer Sciences University of California, Berkeley
Abstract
Training time on large datasets for deep neural networks is the principal workflow bottleneck in a number of important applications of deep learning, such as object classification and detection in automatic driver assistance systems (ADAS). To minimize training time, the training of a deep neural network must be scaled beyond a single machine to as many machines as possible by distributing the optimization method used for training. While a number of approaches have been proposed for distributed stochastic gradient descent (SGD), at the current time synchronous approaches to distributed SGD appear to be showing the greatest performance at large scale. Synchronous scaling of SGD suffers from the need to synchronize all processors on each gradient step and is not resilient in the face of failing or lagging processors. In asynchronous approaches using parameter servers, training is slowed by contention to the parameter server. In this paper we compare the convergence of synchronous and asynchronous SGD for training a modern ResNet network architecture on the ImageNet classification problem. We also propose an asynchronous method, gossiping SGD, that aims to retain the positive features
中文速览
大规模深度神经网络训练（如自动驾驶场景下的目标检测）面临着如何把单机训练高效扩展到数百台机器的难题，核心矛盾在于：同步梯度下降（synchronous SGD）虽然在大规模节点上表现稳定，但每一步都要等最慢的机器；异步方法（asynchronous SGD）虽然不用等待，却因为参数服务器的通信瓶颈而拖慢速度。为此，作者提出了一种名为"gossip SGD"的新方法——用去中心化的"流言传播"聚合算法替代同步训练中的全局归约操作，让每个节点只随机与另一个节点交换参数，从而兼顾异步的灵活性与同步的收敛质量。实验在ImageNet图像分类任务上用ResNet-18进行了系统对比，结果显示：在节点数较少（约32个以内）时，异步方法（包括弹性平均SGD和gossip SGD）收敛更快；而在更大规模（约100个节点）时，同步all-reduce SGD反而更胜一筹。这一发现为不同计算资源规模下选择分布式训练策略提供了直接的实践指导，也为去中心化异步训练方法的设计与理论分析提供了新思路。
原文 arXiv:1611.04581;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.04581v1