SoK: Towards the Science of Security and Privacy in Machine Learning
Nicolas Papernot1, Patrick McDaniel1, Arunesh Sinha2, and Michael Wellman2 1 Pennsylvania State University 2 University of Michigan
Abstract
Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics. ML is now pervasive—new systems and models are being deployed in every domain imaginable, leading to rapid and widespread deployment of software based inference and decision making. There is growing recognition that ML exposes new vulnerabilities in software systems, yet the technical community’s understanding of the nature and extent of these vulnerabilities remains limited. We systematize recent findings on ML security and privacy, focusing on attacks identified on these systems and defenses crafted to date. We articulate a comprehensive threat model for ML, and categorize attacks and defenses within an adversarial framework. Key insights resulting from works both in the ML and security communities are identified and the effectiveness of approaches are related to structural elements of ML algorithms and the data used to train them. We conclude by formally exploring the opposing relationship between model accuracy and resilience to adversarial manipulation. Through these explorations, we show that there are (possibly
中文速览
机器学习系统正在被广泛部署于医疗、金融、安全等各类关键领域，但其内在的安全与隐私漏洞至今缺乏系统性认识。这篇论文对机器学习安全与隐私领域的已有研究进行了系统性梳理（Systematization of Knowledge），提出了一套统一的威胁模型，将攻击与防御按对抗框架分类整理，涵盖训练阶段的数据投毒、推理阶段的对抗样本，以及针对模型机密性的提取攻击等多个方向，并对应归纳了差分隐私、鲁棒训练等防御手段。研究最终给出了一个"没有免费午餐"定理，从理论上证明模型复杂度、预测精度与对抗鲁棒性之间存在内在张力，三者无法同时最优。这项工作为跨越机器学习、安全、统计等多个学科的研究者提供了共同的语言框架，对于如何在实际部署中权衡模型性能与安全性具有重要的指导意义。
原文 arXiv:1611.03814;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.03814v1