Petuum: A New Platform for Distributed Machine Learning on Big Data
Eric P. Xing School of Computer Science, Carnegie Mellon University Qirong Ho Institute for Infocomm Research, A*STAR, Singapore Wei Dai School of Computer Science, Carnegie Mellon University Jin Kyu Kim School of Computer Science, Carnegie Mellon University Jinliang Wei School of Computer Science, Carnegie Mellon University Seunghak Lee School of Computer Science, Carnegie Mellon University Xun Zheng School of Computer Science, Carnegie Mellon University Pengtao Xie School of Computer Science, Carnegie Mellon University Abhimanu Kumar School of Computer Science, Carnegie Mellon University Yaoliang Yu School of Computer Science, Carnegie Mellon University
Abstract
What is a systematic way to efficiently apply a wide spectrum of advanced ML programs to industrial scale problems, using Big Models (up to 100s of billions of parameters) on Big Data (up to terabytes or petabytes)? Modern parallelization strategies employ fine-grained operations and scheduling beyond the classic bulk-synchronous processing paradigm popularized by MapReduce, or even specialized graph-based execution that relies on graph representations of ML programs. The variety of approaches tends to pull systems and algorithms design in different directions, and it remains difficult to find a universal platform applicable to a wide range of ML programs at scale. We propose a general-purpose framework that systematically addresses data- and model-parallel challenges in large-scale ML, by observing that many ML programs are fundamentally optimization-centric and admit error-tolerant, iterative-convergent algorithmic solutions. This presents unique opportunities for an integrative system design, such as bounded-error network synchronization and dynamic scheduling based on ML program structure. We demonstrate the efficacy of these system designs versus well-known implementations of
中文速览
大规模机器学习在工业级应用中面临两大瓶颈：数据量动辄TB级以上（Big Data），模型参数多达数百亿（Big Model），现有平台（如Hadoop、Spark、GraphLab）要么编程抽象太粗糙、要么缺乏细粒度调度、要么不能通用覆盖多种算法，难以同时兼顾效率、正确性和易用性。Petuum框架从一个统一视角切入：绝大多数机器学习算法本质上都是以目标函数为核心的迭代收敛过程，天然具备误差容忍、参数依赖动态变化、收敛速度非均匀这三个统计特性，因此可以把数据并行与模型并行系统性地结合起来，并据此设计出有界延迟（bounded staleness）的网络同步机制、基于动态依赖结构的调度策略，以及优先更新未收敛参数的计算策略。在主题模型、矩阵分解、深度学习、Lasso回归、距离度量学习等多项基准测试中，Petuum在仅十到百台规模的普通集群上便能以更短时间、更大模型规格跑赢主流系统。这项工作的意义在于，它为学术界的ML算法跨越到工业级大规模应用提供了一个理论有据、实践通用的系统基础，让更广泛的开发者无需精通分布式底层细节就能驾驭Big Learning任务。
原文 arXiv:1312.7651;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1312.7651v2