Purine: A bi-graph based deep learning framework
Min Lin1,2, Shuo Li3, Xuan Luo3、Shuicheng Yan2 1. Graduate School of integrated Sciences and Engineering 2. Department of Electrical and Computer Engineering National University of Singapore 3. Zhiyuan College, Shanghai Jiao Tong University {linmin,
Abstract
In this paper, we introduce a novel deep learning framework, termed Purine. In Purine, a deep network is expressed as a bipartite graph (bi-graph), which is composed of interconnected operators and data tensors. With the bi-graph abstraction, networks are easily solvable with event-driven task dispatcher. We then demonstrate that different parallelism schemes over GPUs and/or CPUs on single or multiple PCs can be universally implemented by graph composition. This eases researchers from coding for various parallelization schemes, and the same dispatcher can be used for solving variant graphs. Scheduled by the task dispatcher, memory transfers are fully overlapped with other computations, which greatly reduces the communication overhead and helps us achieve approximate linear acceleration.
中文速览
深度学习训练越来越依赖多GPU、多机器的并行计算，但如何灵活、高效地实现模型并行、数据并行等各种方案一直是个编程难题。Purine框架把神经网络抽象成一种二部图（bipartite graph），图中的节点分为"数据张量"和"算子"两类，并通过一个事件驱动的任务调度器来自动执行；不同的并行策略只需给节点贴上"位置"标签、再做图的组合即可实现，无需为每种方案单独写并行代码。实验在GoogLeNet上验证，使用12块GPU时仍能达到接近线性的加速比，关键在于任务调度器让参数的跨设备传输与反向传播计算完全重叠，大幅降低了通信开销。这项工作为研究者提供了一套统一、简洁的并行深度学习训练基础架构，有望显著降低大规模分布式训练的开发门槛。
原文 arXiv:1412.6249;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1412.6249v5