Thinking While Moving: Deep Reinforcement Learning with Concurrent Control
Ted Xiao1, Eric Jang1, Dmitry Kalashnikov1, Sergey Levine1,2, Julian Ibarz1, Karol Hausman1 , Alexander Herzog 1Google Brain, 2UC Berkeley, 3X {tedxiao, ejang, dkalashnikov, slevine, julianibarz, Indicates equal contribution.
Abstract
We study reinforcement learning in settings where sampling an action from the policy must be done concurrently with the time evolution of the controlled system, such as when a robot must decide on the next action while still performing the previous action. Much like a person or an animal, the robot must think and move at the same time, deciding on its next action before the previous one has completed. In order to develop an algorithmic framework for such concurrent control problems, we start with a continuous-time formulation of the Bellman equations, and then discretize them in a way that is aware of system delays. We instantiate this new class of approximate dynamic programming methods via a simple architectural extension to existing value-based deep reinforcement learning algorithms. We evaluate our methods on simulated benchmark tasks and a large-scale robotic grasping task where the robot must “think while moving”. Videos are available at https://sites.google.com/view/thinkingwhilemoving.
中文速览
现实世界中的机器人在执行当前动作时必须同步计算下一步该怎么动，而主流深度强化学习（Deep Reinforcement Learning）方法却默认环境会"暂停等待"智能体思考，这一假设在真实场景中根本不成立。为此，研究者从连续时间贝尔曼方程（Bellman equation）出发，推导出一套能感知系统延迟的并发控制（concurrent control）理论框架，并将其转化为一个极简的工程改动——在现有基于价值的深度强化学习算法中，把上一时刻的动作也作为Q函数的额外输入，从而让智能体在"边思考边行动"时依然能做出正确决策。在仿真基准任务和大规模真实机器人抓取实验中，该方法以仅需最小架构修改的代价，让机器人在保持与"暂停等待"基线相当的任务成功率的同时，整体运行速度提升了49%。这项工作为把强化学习从"冻结时间"的仿真环境迁移到真实动态场景提供了清晰的理论依据和可落地的实践路径。
原文 arXiv:2004.06089;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.06089v4