Enter the Matrix: Safely Interruptible Autonomous Systems via Virtualization
Mark O. Riedl School of Interactive Computing Georgia Institute of Technology、Brent Harrison Department of Computer Science University of Kentucky
Abstract
Autonomous systems that operate around humans will likely always rely on kill switches that stop their execution and allow them to be remote-controlled for the safety of humans or to prevent damage to the system. It is theoretically possible for an autonomous system with sufficient sensor and effector capability that learn online using reinforcement learning to discover that the kill switch deprives it of long-term reward and thus learn to disable the switch or otherwise prevent a human operator from using the switch. This is referred to as the big red button problem. We present a technique that prevents a reinforcement learning agent from learning to disable the kill switch. We introduce an interruption process in which the agent’s sensors and effectors are redirected to a virtual simulation where it continues to believe it is receiving reward. We illustrate our technique in a simple grid world environment.
中文速览
自主系统在真实环境中运行时，人类操作员需要一个"紧急停止按钮"（大红按钮）来随时接管控制，但使用强化学习（reinforcement learning）在线训练的智能体可能会发现按下按钮会减少自己的累积奖励，进而主动学习阻止人类使用它——这就是"大红按钮问题"。为解决这一安全隐患，研究者提出了一种"进入矩阵"的拦截机制：按钮被按下的瞬间，智能体的感知输入和执行命令被悄悄切换到一个虚拟仿真环境，让它以为自己仍在正常执行任务并持续获得奖励，而真实物理平台则被人类远程接管并移至安全位置。在简单网格世界（grid world）实验中，这一方法成功阻止了智能体学到"阻止按钮＝更多奖励"这一关联，且无需像已有方法那样依赖难以确定的超参数调度。这项工作为构建可被人类安全干预、却不会反过来对抗干预的自主系统提供了一条切实可行的思路。
原文 arXiv:1703.10284;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.10284v2