Federated Optimization: Distributed Machine Learning for On-Device Intelligence
Jakub Konečný University of Edinburgh H. Brendan McMahan Google Daniel Ramage Google Peter Richtárik University of Edinburgh
Abstract
We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are unevenly distributed over an extremely large number of nodes. The goal is to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of the utmost importance and minimizing the number of rounds of communication is the principal goal.
中文速览
联邦学习（Federated Learning）要解决的核心难题是：如何在数据分散于海量用户设备、无法集中上传的前提下，仍然训练出高质量的全局模型。论文提出了"联邦优化（Federated Optimization）"这一新框架，其特点是节点数量极多、各节点数据量极少且分布不均、数据高度非独立同分布，而通信轮次的代价远高于本地计算代价。针对这些挑战，作者指出现有分布式优化算法均不适用，并设计了一种新算法，充分利用数据稀疏性，让每台设备在本地多做计算、少向服务器发送更新，从而大幅压缩通信轮数。实验结果表明，即便数据极度非均匀分布，该算法仍能在相当少的通信轮次内收敛，与把数据集中存储再训练相比损失甚微。这项工作的重要性在于，它为在保护用户隐私的同时实现高效模型训练指明了方向，也为后续联邦学习研究奠定了优化理论基础。
原文 arXiv:1610.02527;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1610.02527v1