SparkNet: Training Deep Networks in Spark
Philipp Moritz, Robert Nishihara∗, Ion Stoica, Michael I. Jordan Electrical Engineering and Computer Science University of California Berkeley, CA 94720, USA Both authors contributed equally.
Abstract
Training deep networks is a time-consuming process, with networks for object recognition often requiring multiple days to train. For this reason, leveraging the resources of a cluster to speed up training is an important area of work. However, widely-popular batch-processing computational frameworks like MapReduce and Spark were not designed to support the asynchronous and communication-intensive workloads of existing distributed deep learning systems. We introduce SparkNet, a framework for training deep networks in Spark. Our implementation includes a convenient interface for reading data from Spark RDDs, a Scala interface to the Caffe deep learning framework, and a lightweight multi-dimensional tensor library. Using a simple parallelization scheme for stochastic gradient descent, SparkNet scales well with the cluster size and tolerates very high-latency communication. Furthermore, it is easy to deploy and use with no parameter tuning, and it is compatible with existing Caffe models. We quantify the dependence of the speedup obtained by SparkNet on the number of machines, the communication frequency, and the cluster’s communication overhead, and we benchmark our system’s performan
中文速览
深度网络训练往往要耗费数天，而常用的 Spark 等批处理框架又不适合频繁通信、异步更新的分布式训练。SparkNet 把 Spark 与 Caffe 结合起来，让各机器在本地连续进行一段时间的随机梯度下降（SGD），再定期汇总并平均模型参数，从而大幅减少通信，同时兼容现有 Caffe 模型和 Spark 数据处理流程。实验表明，它能随机器数量增加获得良好加速，在通信延迟很高、带宽有限的集群上也能稳定工作，并在 ImageNet 上取得接近专用分布式系统的训练表现。它的重要性在于无需复杂调参或专门硬件，就能把深度学习训练直接嵌入已有的大规模数据处理流水线。
原文 arXiv:1511.06051;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.06051v4