SparkNet: Training Deep Networks in Spark
Philipp Moritz, Robert Nishihara∗, Ion Stoica, Michael I. Jordan Electrical Engineering and Computer Science University of California Berkeley, CA 94720, USA Both authors contributed equally.
Abstract
Training deep networks is a time-consuming process, with networks for object recognition often requiring multiple days to train. For this reason, leveraging the resources of a cluster to speed up training is an important area of work. However, widely-popular batch-processing computational frameworks like MapReduce and Spark were not designed to support the asynchronous and communication-intensive workloads of existing distributed deep learning systems. We introduce SparkNet, a framework for training deep networks in Spark. Our implementation includes a convenient interface for reading data from Spark RDDs, a Scala interface to the Caffe deep learning framework, and a lightweight multi-dimensional tensor library. Using a simple parallelization scheme for stochastic gradient descent, SparkNet scales well with the cluster size and tolerates very high-latency communication. Furthermore, it is easy to deploy and use with no parameter tuning, and it is compatible with existing Caffe models. We quantify the dependence of the speedup obtained by SparkNet on the number of machines, the communication frequency, and the cluster’s communication overhead, and we benchmark our system’s performan
中文速览
深度神经网络的训练动辄耗时数天，利用集群加速训练是迫切需求，但主流批处理框架（如MapReduce、Spark）并不为异步、高通信量的深度学习负载而设计。SparkNet在Apache Spark和Caffe的基础上构建了一套分布式深度学习训练框架，采用一种极简通信的并行随机梯度下降（SGD）策略：每台工作节点独立跑若干轮本地迭代后，主节点再将各节点模型参数收集并取平均、广播给所有节点，从而将通信频率降低约50倍。在ImageNet上的实测表明，SparkNet能在带宽有限的EC2集群上随机器数量近线性地提升训练速度，无需手动调参，且与现有Caffe模型完全兼容。这项工作的意义在于，它让深度学习训练能无缝融入企业已有的大数据处理流水线，极大降低了分布式训练的部署门槛，即便在通信条件不理想的普通云集群上也能获得实质性的加速效果。
原文 arXiv:1511.06051;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.06051v4