Differential Privacy Guarantees for Stochastic Gradient Langevin Dynamics
Théo Ryffel Francis Bach David Pointcheval
Abstract
We analyse the privacy leakage of noisy stochastic gradient descent by modeling Rényi divergence dynamics with Langevin diffusions. Inspired by recent work on non-stochastic algorithms, we derive similar desirable properties in the stochastic setting. In particular, we prove that the privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size, which is a significant improvement over previous DP-SGD analyses. We also extend our analysis to arbitrary sequences of varying step sizes and derive new utility bounds. Last, we propose an implementation and our experiments show the practical utility of our approach compared to classical DP-SGD libraries.
中文速览
差分隐私随机梯度下降（DP-SGD）在训练深度学习模型时面临一个棘手问题：每多训练一步，隐私泄露就会累积增长，导致大量迭代后隐私保证几乎失效。这篇论文借鉴最近针对全量梯度下降（DP-GLD）的朗之万扩散（Langevin diffusion）分析框架，提出了适用于小批量随机梯度的 DP-SGLD 算法，并通过建模 Rényi 散度的动态演化，证明在目标函数光滑且强凸的条件下，隐私损失会以指数速度收敛到一个有界常数，彻底打破了"训练越多、隐私越差"的瓶颈。此外，论文还将分析推广到变步长情形，并给出了新的模型效用界，实验结果表明 DP-SGLD 在多个数据集的逻辑回归任务上显著优于经典 DP-SGD，几乎追平非隐私训练的精度。这项工作使得在只公开最终模型（如联邦学习或安全多方计算场景）的条件下，可以安全地进行任意多轮训练，具有重要的实际意义。
原文 arXiv:2201.11980;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2201.11980v2