Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
William Saunders University of Oxford、Girish Sastry University of Oxford、Andreas Stuhlmüller Stanford University、Owain Evans University of Oxford
Abstract
AI systems are increasingly applied to complex tasks that involve interaction with humans. During training, such systems are potentially dangerous, as they haven’t yet learned to avoid actions that could cause serious harm. How can an AI system explore and learn without making a single mistake that harms humans or otherwise causes serious damage? For model-free reinforcement learning, having a human “in the loop” and ready to intervene is currently the only way to prevent all catastrophes. We formalize human intervention for RL and show how to reduce the human labor required by training a supervised learner to imitate the human’s intervention decisions. We evaluate this scheme on Atari games, with a Deep RL agent being overseen by a human for four hours. When the class of catastrophes is simple, we are able to prevent all catastrophes without affecting the agent’s learning (whereas an RL baseline fails due to catastrophic forgetting). However, this scheme is less successful when catastrophes are more complex: it reduces but does not eliminate catastrophes and the supervised learner fails on adversarial examples found by the agent. Extrapolating to more challenging environments, we
中文速览
训练中的AI智能体因为还没学会避免危险行为，可能在与人类交互时造成严重伤害——这篇论文要解决的就是"如何让强化学习（Reinforcement Learning, RL）智能体在训练过程中一次灾难性错误都不犯"的问题。研究者提出了一套名为HIRL（Human Intervention RL，人类干预强化学习）的方案：先让人类全程监督智能体的训练，随时拦截危险动作，同时收集这些干预数据来训练一个"拦截器"（Blocker）分类模型，再让这个模型自动接管人类的监督工作。他们在Atari游戏上验证了该方案，当灾难类型比较简单时（如Pong、Space Invaders），拦截器能做到零失误，而纯靠负奖励的RL基线方法则因"灾难性遗忘"始终无法彻底消除错误；但在更复杂的场景（Road Runner）中，拦截器面对智能体找到的对抗样本会失效，灾难次数只能减少而无法归零。这项工作的重要意义在于，它正视了将人类监督引入深度RL时面临的真实瓶颈——人力成本不可扩展、拦截器分布偏移等问题——并为未来构建真正安全的AI训练流程指明了需要突破的方向。
原文 arXiv:1707.05173;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.05173v1