Reinforcement and Imitation Learning via Interactive No-Regret Learning
Stéphane Ross J. Andrew Bagnell The Robotics Institute Carnegie Mellon University, Pittsburgh, PA, USA
Abstract
Recent work has demonstrated that problems– particularly imitation learning and structured prediction– where a learner’s predictions influence the input-distribution it is tested on can be naturally addressed by an interactive approach and analyzed using no-regret online learning. These approaches to imitation learning, however, neither require nor benefit from information about the cost of actions. We extend existing results in two directions: first, we develop an interactive imitation learning approach that leverages cost information; second, we extend the technique to address reinforcement learning. The results provide theoretical support to the commonly observed successes of online approximate policy iteration. Our approach suggests a broad new family of algorithms and provides a unifying view of existing techniques for imitation and reinforcement learning.
中文速览
模仿学习领域长期存在一个痛点：现有方法（如DAgger）只关心"跟专家动作是否一致"，却忽视了不同错误的代价差异——比如开车偏向悬崖一点点和直接冲下去，在它们眼里是等价的错误。本文提出AggreVaTe算法，在交互式模仿学习的框架中引入"代价到达终点的期望损失"（cost-to-go）信息：每轮迭代时，让当前策略执行到某个随机时刻，探索一个动作，再交还给专家继续完成任务，从而估算该动作的长期代价，并用无悔（no-regret）在线学习算法持续改进策略。理论上，只要底层学习器满足无悔性质，AggreVaTe就能给出比现有方法更强的统计保证（统计遗憾而非统计误差缩减），且该框架自然延伸到近似策略迭代（Approximate Policy Iteration），为强化学习中在线策略迭代的经验成功提供了理论支撑。这项工作统一了模仿学习与强化学习中多种已有算法，并为设计新算法提供了清晰的原则性框架。
原文 arXiv:1406.5979;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1406.5979v1