Fast is better than free: Revisiting adversarial training
Eric Wong Machine Learning Department Carnegie Mellon University Pittsburgh, PA 15213, USA、Leslie Rice Computer Science Department Carnegie Mellon University Pittsburgh, PA 15213, USA、J. Zico Kolter Computer Science Department Carnegie Mellon University and Bosch Center for Artifical Intelligence Pittsburgh, PA 15213, USA Equal contribution.
Abstract
Adversarial training, a method for learning robust deep networks, is typically assumed to be more expensive than traditional training due to the necessity of constructing adversarial examples via a first-order method like projected gradient decent (PGD). In this paper, we make the surprising discovery that it is possible to train empirically robust models using a much weaker and cheaper adversary, an approach that was previously believed to be ineffective, rendering the method no more costly than standard training in practice. Specifically, we show that adversarial training with the fast gradient sign method (FGSM), when combined with random initialization, is as effective as PGD-based training but has significantly lower cost. Furthermore we show that FGSM adversarial training can be further accelerated by using standard techniques for efficient training of deep networks, allowing us to learn a robust CIFAR10 classifier with 45% robust accuracy to PGD attacks with $\epsilon=8/255$ in 6 minutes, and a robust ImageNet classifier with 43% robust accuracy at $\epsilon=2/255$ in 12 hours, in comparison to past work based on “free” adversarial training which took 10 and 50 hours to reac
中文速览
对抗训练（adversarial training）长期被认为比普通训练慢得多，因为需要用多步梯度攻击（PGD）来生成对抗样本，训练一个鲁棒的 CIFAR10 分类器就要花费约 80 小时。本文发现，只需在单步快速梯度符号法（FGSM）之前加入随机初始化这一简单改动，FGSM 对抗训练就能达到与 PGD 对抗训练相当的鲁棒性，同时还能结合循环学习率、混合精度等常见加速技巧，将训练时间压缩到与普通训练相当的量级。实验结果显示，在单张 GPU 上，鲁棒 CIFAR10 模型只需 6 分钟、鲁棒 ImageNet 模型只需 12 小时即可训练完成，比此前最快的"免费"对抗训练方法分别快约 100 倍和 4 倍。此外，论文还揭示了一种名为"灾难性过拟合"（catastrophic overfitting）的失效模式，解释了为何早期 FGSM 对抗训练会失败，这一发现对推动大规模鲁棒深度学习研究具有重要意义。
原文 arXiv:2001.03994;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2001.03994v1