Model Accuracy and Runtime Tradeoff in Distributed Deep Learning: A Systematic Study
Suyog Gupta1,∗, Wei Zhang1,∗, Fei Wang2 1IBM T. J. Watson Research Center. Yorktown Heights. NY. ∗Equal Contribution. 2Department of Healthcare Policy and Research, Weill Cornell Medical College. New York City. NY
Abstract
Deep learning with a large number of parameters requires distributed training, where model accuracy and runtime are two important factors to be considered. However, there has been no systematic study of the tradeoff between these two factors during the model training process. This paper presents Rudra, a parameter server based distributed computing framework tuned for training large-scale deep neural networks. Using variants of the asynchronous stochastic gradient descent algorithm we study the impact of synchronization protocol, stale gradient updates, minibatch size, learning rates, and number of learners on runtime performance and model accuracy. We introduce a new learning rate modulation strategy to counter the effect of stale gradients and propose a new synchronization protocol that can effectively bound the staleness in gradients, improve runtime performance and achieve good model accuracy. Our empirical investigation reveals a principled approach for distributed training of neural networks: the mini-batch size per learner should be reduced as more learners are added to the system to preserve the model accuracy. We validate this approach using commonly-used image classificat
中文速览
大规模深度神经网络训练既要跑得快、又要准确率高，但这两个目标之间如何权衡此前缺乏系统研究。作者构建了一个名为 Rudra 的参数服务器（parameter server）分布式训练框架，围绕异步随机梯度下降（asynchronous SGD）系统地考察了同步协议、梯度过时程度（gradient staleness）、小批量大小（mini-batch size）、学习率和并行节点数量对训练速度与模型精度的联合影响。研究发现，随着并行节点增多，每个节点的小批量应相应减小，才能维持模型精度，这意味着数据并行的可扩展性存在实际上限；与此同时，作者提出了一种基于过时程度动态调整学习率的策略，以及一种新的"软同步"协议，在有效控制梯度过时、降低网络带宽压力的同时保持良好精度。上述结论在 CIFAR-10 和 ImageNet 图像分类基准上得到了验证，为工业界大规模分布式深度学习提供了可操作的系统调优原则。
原文 arXiv:1509.04210;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1509.04210v3