The Bottleneck Simulator: A Model-based Deep Reinforcement Learning Approach
Iulian Vlad Serban Chinnadhurai Sankar Michael Pieper Joelle Pineau Yoshua Bengio
Abstract
Deep reinforcement learning has recently shown many impressive successes. However, one major obstacle towards applying such methods to real-world problems is their lack of data-efficiency. To this end, we propose the Bottleneck Simulator: a model-based reinforcement learning method which combines a learned, factorized transition model of the environment with rollout simulations to learn an effective policy from few examples. The learned transition model employs an abstract, discrete (bottleneck) state, which increases sample efficiency by reducing the number of model parameters and by exploiting structural properties of the environment. We provide a mathematical analysis of the Bottleneck Simulator in terms of fixed points of the learned policy, which reveals how performance is affected by four distinct sources of error: an error related to the abstract space structure, an error related to the transition model estimation variance, an error related to the transition model estimation bias, and an error related to the transition model class bias. Finally, we evaluate the Bottleneck Simulator on two natural language processing tasks: a text adventure game and a real-world, complex dial
中文速览
深度强化学习在样本效率上存在严重短板——真实场景中很难收集到海量交互数据。为解决这一问题，研究者提出了"瓶颈模拟器"（Bottleneck Simulator）：先用少量数据学习一个分解式环境转移模型，其核心在于引入离散的抽象状态作为信息瓶颈，将相似的真实状态归为同一抽象状态，从而大幅削减模型参数、降低对数据量的需求；再在这个近似模型上通过滚动仿真（rollout simulation）学习策略。研究还给出了严格的数学分析，将策略误差分解为四个来源：抽象空间结构误差、转移模型估计方差、估计偏差以及模型类偏差，清晰揭示了各因素如何影响最终性能。在文本冒险游戏和真实对话回复选择这两项自然语言处理任务上，瓶颈模拟器均以少量样本超越了对比方法。这项工作的意义在于，它提供了一条将模型化强化学习推向数据稀缺的真实应用场景的可行路径，并配以可解释的理论保证。
原文 arXiv:1807.04723;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1807.04723v1