Precise Tradeoffs in Adversarial Training for Linear Regression
Adel Javanmard, Mahdi Soltanolkotabi, Hamed Hassani Data Science and Operations Department, Marshall School of Business, University of Southern California, Los Angeles, CAMing Hsieh Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CADepartment of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA
Abstract
Despite breakthrough performance, modern learning models are known to be highly vulnerable to small adversarial perturbations in their inputs. While a wide variety of recent adversarial training methods have been effective at improving robustness to perturbed inputs (robust accuracy), often this benefit is accompanied by a decrease in accuracy on benign inputs (standard accuracy), leading to a tradeoff between often competing objectives. Complicating matters further, recent empirical evidence suggest that a variety of other factors (size and quality of training data, model size, etc.) affect this tradeoff in somewhat surprising ways. In this paper we provide a precise and comprehensive understanding of the role of adversarial training in the context of linear regression with Gaussian features. In particular, we characterize the fundamental tradeoff between the accuracies achievable by any algorithm regardless of computational power or size of the training data. Furthermore, we precisely characterize the standard/robust accuracy and the corresponding tradeoff achieved by a contemporary mini-max adversarial training approach in a high-dimensional regime where the number of data point
中文速览
深度学习模型容易被微小的对抗性扰动"骗过"，而对抗训练（adversarial training）虽然能提升模型在被攻击输入上的鲁棒精度，却往往损害其在正常输入上的标准精度，二者之间存在内在张力。为了从理论上彻底搞清楚这一矛盾，研究者聚焦于高斯特征线性回归这一基础模型，借助凸高斯极小极大定理（Convex Gaussian Minimax Theorem）等工具，在样本数与参数数等比例增长的高维渐近框架下，给出了标准风险与对抗风险之间权衡曲线的精确解析刻画——既包括任何算法都无法突破的帕累托最优基本边界，也包括当代极小极大对抗训练算法实际能达到的权衡曲线。理论结果揭示了几个关键规律：在过参数化（overparametrization）区间，对抗训练反而有助于改善标准泛化误差；随着训练数据增多进入欠参数化区间，对抗训练则会损害标准精度；此外，对抗训练场景下同样存在类似"双重下降"（double-descent）的现象，但插值阈值会随对抗扰动强度的变化而移动。这项工作为理解对抗训练中数据量、模型规模、扰动强度等因素如何共同影响鲁棒性与准确性之间的取舍提供了严格的理论依据。
原文 arXiv:2002.10477;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.10477v1