PipeDream: Fast and Efficient Pipeline Parallel DNN Training
Aaron Harlap† Deepak Narayanan‡ ††Amar Phanishayee⋆ Vivek Seshadri⋆ Nikhil Devanur⋆ Greg Ganger† Phil Gibbons† ⋆Microsoft Research †Carnegie Mellon University ‡Stanford University Work started as part of an internship at Microsoft Research.
Abstract
PipeDream is a Deep Neural Network (DNN) training system for GPUs that parallelizes computation by pipelining execution across multiple machines. Its pipeline parallel computing model avoids the slowdowns faced by data-parallel training when large models and/or limited network bandwidth induce high communication-to-computation ratios. PipeDream reduces communication by up to 95% for large DNNs relative to data-parallel training, and allows perfect overlap of communication and computation. PipeDream keeps all available GPUs productive by systematically partitioning DNN layers among them to balance work and minimize communication, versions model parameters for backward pass correctness, and schedules the forward and backward passes of different inputs in round-robin fashion to optimize “time to target accuracy”. Experiments with five different DNNs on two different clusters show that PipeDream is up to 5x faster in time-to-accuracy compared to data-parallel training.
中文速览
数据并行训练（data-parallel training）在面对参数量庞大的深度神经网络（DNN）时，通信开销会严重拖慢训练速度——在某些模型和硬件组合下，通信时间甚至占到总训练时间的85%，而随着GPU算力持续提升，这一问题只会越来越严峻。PipeDream提出了一种"流水线并行"（pipeline-parallel）训练机制，将模型按层切分成多个阶段分配给不同的机器，同时向流水线持续注入多个小批次数据，让所有机器始终保持忙碌；对于无法整除的阶段，系统还自动引入数据并行来均衡负载，并通过保留多个参数版本确保反向传播使用前向传播时一致的权重，从而不损害模型收敛质量。与数据并行的同步训练（BSP）相比，PipeDream将机器间通信量最多削减95%，在五种主流DNN模型（包括VGG16、AlexNet、Inception-v3、ResNet-50和S2VT）上实现了最高5倍以上的训练加速。这项工作的意义在于，它以系统化、自动化的方式打通了流水线、模型并行与数据并行三者的结合，为大规模DNN训练在通信受限场景下提供了一条切实可行的出路。
原文 arXiv:1806.03377;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1806.03377v1