Discrete Sequential Prediction of Continuous Actions for Deep RL
Luke Metz Google Brain、Julian Ibarz Google Brain、Navdeep Jaitly Google Brain、James Davidson Google Brain
Abstract
It has long been assumed that high dimensional continuous control problems cannot be solved effectively by discretizing individual dimensions of the action space due to the exponentially large number of bins over which policies would have to be learned. In this paper, we draw inspiration from the recent success of sequence-to-sequence models for structured prediction problems to develop policies over discretized spaces. Central to this method is the realization that complex functions over high dimensional spaces can be modeled by neural networks that predict one dimension at a time. Specifically, we show how Q-values and policies over continuous spaces can be modeled using a next step prediction model over discretized dimensions. With this parameterization, it is possible to both leverage the compositional structure of action spaces during learning, as well as compute maxima over action spaces (approximately). On a simple example task we demonstrate empirically that our method can perform global search, which effectively gets around the local optimization issues that plague DDPG. We apply the technique to off-policy (Q-learning) methods and show that our method can achieve the stat
中文速览
连续动作空间的强化学习一直面临一个两难困境：直接离散化每个动作维度会导致动作数量指数爆炸，而不离散化又很难像 DQN 那样高效地做全局最优搜索。这篇论文借鉴序列到序列（sequence-to-sequence）模型的思路，提出了 Sequential DQN（SDQN）——把一个 N 维的连续动作拆解成 N 步依次预测的一维动作序列，每一维单独离散化，用自回归方式建模 Q 值，从而在不爆炸参数量的前提下实现细粒度的动作离散化。通过在上层 MDP 和下层序列 MDP 之间共享 Q 值并做 Bellman 备份，模型既能近似求解全局最优动作，又避免了 DDPG 依赖确定性策略梯度所带来的局部最优陷阱。实验表明，SDQN 在多峰值测试任务上能有效做全局搜索，并在 hopper 到 humanoid 等多个连续控制基准上达到了当时 off-policy 方法的最优水平，为连续控制问题提供了一条将离散 Q-learning 优势引入高维连续空间的可行路径。
原文 arXiv:1705.05035;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.05035v3