MLlib: Machine Learning in Apache Spark
\nameXiangrui Meng††\dagger \addrDatabricks, 160 Spear Street, 13th Floor, San Francisco, CA 94105 \AND\nameJoseph Bradley \addrDatabricks, 160 Spear Street, 13th Floor, San Francisco, CA 94105 \AND\nameBurak Yavuz \addrDatabricks, 160 Spear Street, 13th Floor, San Francisco, CA 94105 \AND\nameEvan Sparks \addrUC Berkeley, 465 Soda Hall, Berkeley, CA 94720 \AND\nameShivaram Venkataraman \addrUC Berkeley, 465 Soda Hall, Berkeley, CA 94720 \AND\nameDavies Liu \addrDatabricks, 160 Spear Street, 13th Floor, San Francisco, CA 94105 \AND\nameJeremy Freeman \addrHHMI Janelia Research Campus, 19805 Helix Dr, Ashburn, VA 20147 \AND\nameDB Tsai \addrNetflix, 970 University Ave, Los Gatos, CA 95032 \AND\nameManish Amde \addrOrigami Logic, 1134 Crane Street, Menlo Park, CA 94025 \AND\nameSean Owen \addrCloudera UK, 33 Creechurch Lane, London EC3A 5EB United Kingdom \AND\nameDoris Xin \addrUIUC, 201 N Goodwin Ave, Urbana, IL 61801 \AND\nameReynold \addrDatabricks, 160 Spear Street, 13th Floor, San Francisco, CA 94105 \AND\nameMichael J. Franklin \addrUC Berkeley, 465 Soda Hall, Berkeley, CA 94720 \AND\nameReza Zadeh \addrStanford and Databricks, 475 Via Ortega, Stanford, CA 94305 \AND\nameMatei Zaharia \addrMIT and Databricks, 160 Spear Street, 13th Floor, San Francisco, CA 94105 \AND\nameAmeet Talwalkar††\dagger \addrUCLA and Databricks, 4732 Boelter Hall, Los Angeles, CA 90095
Abstract
Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark’s open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark’s rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.
中文速览
大规模机器学习在工业界落地时面临算法分散、流水线搭建繁琐、迭代计算效率低下等难题，MLlib 正是为解决这些问题而生。它作为 Apache Spark 的内置分布式机器学习库，将分类、回归、协同过滤、聚类、降维等常用算法统一集成，并借助 Spark 对迭代计算的原生支持和 C++ 底层线性代数库来提升运行效率，同时提供 spark.ml 流水线（pipeline）API，让数据预处理、特征提取、模型训练到验证的全流程开发变得简洁一致。基准测试表明，MLlib 在相同规模数据集上远快于基于 Hadoop MapReduce 的 Apache Mahout，且从 1.0 到 1.1 版本跨算法平均提速达 3 倍。MLlib 由超过 140 名来自 50 余家机构的贡献者共同维护，已成为当前规模最大的开源分布式机器学习库，为工业界和学术界在海量数据上快速构建端到端机器学习系统提供了坚实基础。
原文 arXiv:1505.06807;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1505.06807v1