Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks
Vahid Behzadan Arslan Munir
Abstract
Deep learning classifiers are known to be inherently vulnerable to manipulation by intentionally perturbed inputs, named adversarial examples. In this work, we establish that reinforcement learning techniques based on Deep Q-Networks (DQNs) are also vulnerable to adversarial input perturbations, and verify the transferability of adversarial examples across different DQN models. Furthermore, we present a novel class of attacks based on this vulnerability that enable policy manipulation and induction in the learning process of DQNs. We propose an attack mechanism that exploits the transferability of adversarial examples to implement policy induction attacks on DQNs, and demonstrate its efficacy and impact through experimental study of a game-learning scenario.
中文速览
深度强化学习中的Q网络（Deep Q-Network, DQN）被广泛用于游戏、自动驾驶等高风险场景，但其安全性此前未受到充分审视。研究者证明DQN与图像分类器一样容易被对抗样本（adversarial examples）所欺骗——只需在环境观测输入（如游戏画面像素）上施加人眼几乎察觉不到的微小扰动，就能操控智能体的决策策略。在此基础上，他们提出了一种"策略诱导攻击"（policy induction attack）方法：攻击者无需了解目标DQN的内部结构，只需训练一个替代网络，利用对抗样本在不同模型间可迁移的特性，便能在学习过程中将任意恶意策略强加给目标智能体。在Atari游戏学习实验中，该攻击以极小的扰动代价成功篡改了DQN所学习的行为策略，且现有的防御蒸馏和对抗训练等手段均不足以有效抵御。这一发现揭示了将DQN部署于无人机导航等安全关键系统时的潜在风险，并呼吁学界重视强化学习系统的对抗鲁棒性问题。
原文 arXiv:1701.04143;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1701.04143v1