Initialization Matters for Adversarial Transfer Learning
Andong Hua1 Jindong Gu2 Zhiyu Xue1 Nicholas Carlini3 Eric Wong4 Yao Qin1,3 1University of California, Santa Barbara 2University of Oxford 3Google 4University of Pennsylvania
Abstract
With the prevalence of the Pretraining-Finetuning paradigm in transfer learning, the robustness of downstream tasks has become a critical concern. In this work, we delve into adversarial robustness in transfer learning and reveal the critical role of initialization, including both the pretrained model and the linear head. First, we discover the necessity of an adversarially robust pretrained model. Specifically, we reveal that with a standard pretrained model, Parameter-Efficient Finetuning (PEFT) methods either fail to be adversarially robust or continue to exhibit significantly degraded adversarial robustness on downstream tasks, even with adversarial training during finetuning. Leveraging a robust pretrained model, surprisingly, we observe that a simple linear probing can outperform full finetuning and other PEFT methods with random initialization on certain datasets. We further identify that linear probing excels in preserving robustness from the robust pretraining. Based on this, we propose Robust Linear Initialization (RoLI) for adversarial finetuning, which initializes the linear head with the weights obtained by adversarial linear probing to maximally inherit the robustness
中文速览
预训练-微调（Pretraining-Finetuning）范式已广泛应用于迁移学习，但下游任务的对抗鲁棒性仍是一大难题。研究者系统考察了六种主流微调方法，发现"初始化"是决定对抗鲁棒性的核心因素：若以标准预训练模型作为起点，即便在微调阶段引入对抗训练，参数高效微调（Parameter-Efficient Finetuning, PEFT）方法也几乎无法获得可用的鲁棒性，即使换用更大规模的 CLIP 预训练模型也无济于事；而以对抗鲁棒预训练模型初始化则能从根本上扭转这一局面。更出人意料的是，在具备鲁棒预训练模型的前提下，简单的线性探测（linear probing）在部分数据集上反而优于全量微调，原因在于它能最大程度地保留预训练阶段积累的鲁棒性。基于这一发现，研究者提出了"鲁棒线性初始化"（Robust Linear Initialization, RoLI）方法——先通过对抗线性探测获得分类头权重，再将其作为后续对抗微调的初始化，从而在继承鲁棒性的同时保持对目标域的特征适配能力；在五个图像分类数据集上，RoLI 分别将干净准确率和鲁棒准确率平均提升了 3.88% 和 2.44%，刷新了对抗迁移学习的最优基准。
原文 arXiv:2312.05716;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2312.05716v2