Visual Prompting for Adversarial Robustness
Abstract
In this work, we leverage visual prompting (VP) to improve adversarial robustness of a fixed, pre-trained model at test time. Compared to conventional adversarial defenses, VP allows us to design universal (i.e., data-agnostic) input prompting templates, which have plug-and-play capabilities at test time to achieve desired model performance without introducing much computation overhead. Although VP has been successfully applied to improving model generalization, it remains elusive whether and how it can be used to defend against adversarial attacks. We investigate this problem and show that the vanilla VP approach is not effective in adversarial defense since a universal input prompt lacks the capacity for robust learning against sample-specific adversarial perturbations. To circumvent it, we propose a new VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate class-wise visual prompts so as to not only leverage the strengths of ensemble prompts but also optimize their interrelations to improve model robustness. Our experiments show that C-AVP outperforms the conventional VP method, with 2.1 $\times$ standard accuracy gain and 2 $\times$ robust accuracy gain
中文速览
针对固定预训练模型在测试阶段难以抵御对抗攻击、而传统对抗训练又计算代价高昂的问题，研究者提出用视觉提示（Visual Prompting, VP）来实现轻量级的测试时防御。他们首先发现，直接用一个"万能"（data-agnostic）的通用视觉提示来对抗攻击效果很差，因为单一的全局提示根本无法应对样本各异的对抗扰动；为此，他们设计了一种名为类别感知对抗视觉提示（Class-wise Adversarial Visual Prompting, C-AVP）的新方法，为每个类别单独学习一个视觉提示，同时通过联合优化损失函数来解决测试时提示选错和"错误提示充当后门触发器"这两个关键难题。在CIFAR-10上的实验表明，C-AVP相比通用版本分别取得了2.1倍的标准准确率提升和2倍的鲁棒准确率提升，推理速度更比经典测试时防御方法快42倍。这项工作的价值在于，它开辟了一条无需重新训练模型、计算开销极小却能有效提升对抗鲁棒性的新路径，为大规模部署场景下的安全防护提供了实用方案。
原文 arXiv:2210.06284;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.06284v4