Parallel SGD: When does averaging help?
Jian Zhang, Christopher De Sa, Ioannis Mitliagkas, and Christopher Ré Department of Computer Science, Stanford University Department of Statistics, Stanford University
Abstract
Consider a number of workers running SGD independently on the same pool of data and averaging the models every once in a while — a common but not well understood practice. We study model averaging as a variance-reducing mechanism and describe two ways in which the frequency of averaging affects convergence. For convex objectives, we show the benefit of frequent averaging depends on the gradient variance envelope. For non-convex objectives, we illustrate that this benefit depends on the presence of multiple optimal points. We complement our findings with multicore experiments on both synthetic and real data.
中文速览
多个工人并行跑随机梯度下降（SGD）、隔一段时间平均一次模型，这种做法在实践中很常见，但理论上为何有效、该多久平均一次却缺乏清晰认识。本文把模型平均看作一种降低方差的机制，核心发现是：对凸问题，梯度方差是否随远离最优点而增大（用参数ρ量化）决定了频繁平均能带来多少收益；对非凸问题，一次性平均（one-shot averaging）可能因不同工人收敛到不同局部最优而产生严重误差，增加平均频率能有效避免这一问题。实验在合成数据、稀疏/稠密回归数据集以及卷积神经网络（MNIST）上均验证了上述结论，表明适当提高平均频率不仅能加快收敛，在非凸场景下更是保证结果正确性的关键。这一工作为设计并行SGD系统时如何在通信开销与统计效率之间做权衡提供了有理论依据的实用指导。
原文 arXiv:1606.07365;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.07365v1