On the Linear Convergence of the Alternating Direction Method of Multipliers
Mingyi Hong and Zhi-Quan Luo Dedicated to the fond memories of a close friend and collaborator, Paul Y. Tseng Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, MN 55455, USA. Email: {mhong,
Abstract
We analyze the convergence rate of the alternating direction method of multipliers (ADMM) for minimizing the sum of two or more nonsmooth convex separable functions subject to linear constraints. Previous analysis of the ADMM typically assumes that the objective function is the sum of only two convex functions defined on two separable blocks of variables even though the algorithm works well in numerical experiments for three or more blocks. Moreover, there has been no rate of convergence analysis for the ADMM without strong convexity in the objective function. In this paper we establish the global linear convergence of the ADMM for minimizing the sum of any number of convex separable functions. This result settles a key question regarding the convergence of the ADMM when the number of blocks is more than two or if the strong convexity is absent. It also implies the linear convergence of the ADMM for several contemporary applications including LASSO, Group LASSO and Sparse Group LASSO without any strong convexity assumption. Our proof is based on estimating the distance from a dual feasible solution to the optimal dual solution set by the norm of a certain proximal residual, and by
中文速览
交替方向乘子法（ADMM）是求解带线性约束的可分凸优化问题的流行算法，但此前理论分析只覆盖两个变量块的情形，三块及以上时收敛性是否成立一直是悬而未决的公开问题，且即便是两块情形也缺乏无强凸性假设下的收敛速率结论。本文通过建立一个误差界（error bound）——用近端残差的范数来估计对偶可行解到最优对偶解集的距离，并要求对偶步长足够小，严格证明了任意多个变量块情形下ADMM的全局线性收敛性。这一结果直接推出：在不需要强凸性的条件下，ADMM求解LASSO、Group LASSO、Sparse Group LASSO等一大类压缩感知问题时同样线性收敛。该工作不仅从理论上填补了多块ADMM收敛分析长达数十年的空白，也为大量实际结构化凸优化应用提供了可靠的算法保证。
原文 arXiv:1208.3922;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1208.3922v3