Private GANs, Revisited
Alex Bie University of Waterloo Work performed in part while interning at Huawei. Also supported by an NSERC Discovery Grant, a David R. Cheriton Graduate Scholarship, and an Ontario Graduate Scholarship. Gautam Kamath University of Waterloo Supported by an NSERC Discovery Grant, an unrestricted gift from Google, and a University of Waterloo startup grant. Guojun Zhang Huawei Noah’s Ark Lab
Abstract
We show that the canonical approach for training differentially private GANs – updating the discriminator with differentially private stochastic gradient descent (DPSGD) – can yield significantly improved results after modifications to training. Specifically, we propose that existing instantiations of this approach neglect to consider how adding noise only to discriminator updates inhibits discriminator training, disrupting the balance between the generator and discriminator necessary for successful GAN training. We show that a simple fix – taking more discriminator steps between generator steps – restores parity between the generator and discriminator and improves results.
中文速览
用差分隐私（Differential Privacy, DP）训练生成对抗网络（GAN）时，判别器需要承受梯度裁剪和噪声注入，而生成器却不受影响，这种不对称会严重削弱判别器、打破两者之间的训练平衡，导致以往基于 DPSGD 的私有 GAN（DPGAN）表现很差、被后续工作普遍抛弃。本文指出，只需要一个简单改动——在每次更新生成器之前多更新几步判别器（有时多达 100 步）——就能补偿噪声带来的损失，恢复两者的训练平衡；在此基础上再结合更大的批量和自适应判别器更新频率，效果进一步提升。实验表明，经过这些修正的 DPGAN 在 MNIST、FashionMNIST 等标准图像生成基准上超越了所有其他私有 GAN 方案，与专门为差分隐私设计的定制方法相比也具有竞争力。这一结论颠覆了学界对 DPGAN 固有缺陷的普遍认知，说明直接将 DPSGD 应用于主流生成模型仍是强有力的选择，不必转向结构复杂的专用方案。
原文 arXiv:2302.02936;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2302.02936v2