BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
Tianyu Gu New York University Brooklyn, NY, USA Brendan Dolan-Gavitt New York University Brooklyn, NY, USA Siddharth Garg New York University Brooklyn, NY, USA
Abstract
Deep learning-based techniques have achieved state-of-the-art performance on a wide variety of recognition and classification tasks. However, these networks are typically computationally expensive to train, requiring weeks of computation on many GPUs; as a result, many users outsource the training procedure to the cloud or rely on pre-trained models that are then fine-tuned for a specific task. In this paper we show that outsourced training introduces new security risks: an adversary can create a maliciously trained network (a backdoored neural network, or a BadNet) that has state-of-the-art performance on the user’s training and validation samples, but behaves badly on specific attacker-chosen inputs. We first explore the properties of BadNets in a toy example, by creating a backdoored handwritten digit classifier. Next, we demonstrate backdoors in a more realistic scenario by creating a U.S. street sign classifier that identifies stop signs as speed limits when a special sticker is added to the stop sign; we then show in addition that the backdoor in our US street sign detector can persist even if the network is later retrained for another task and cause a drop in accuracy of 25%
中文速览
深度神经网络训练成本极高，很多人选择把训练任务外包给云服务商或直接下载预训练模型，这一做法却藏着一个隐患——恶意训练方可以在模型里悄悄植入"后门"（backdoor）。研究者提出了"坏网络"（BadNet）攻击：通过在训练数据中混入带有特定触发标记（backdoor trigger）的样本，让模型在正常输入上表现与无辜模型别无二致，却在含有触发标记的输入上按攻击者的意图输出错误结果。实验中，研究者在手写数字识别和美国街道标志识别两个任务上验证了这一攻击：被植入后门的模型可以让贴了特定贴纸的停车标志被识别为限速标志，且对正常图片的精度几乎不受影响；更危险的是，即使用户将这个含后门的模型做迁移学习（transfer learning）再训练到新任务上，后门依然存活，触发时平均导致准确率下降25%。这项研究表明神经网络后门攻击既强大又隐蔽，警示了外包机器学习训练的安全风险，同时呼吁学界加快研究用于验证和"调试"神经网络行为的工具。
原文 arXiv:1708.06733;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1708.06733v2