QMDP-Net: Deep Learning for Planning under Partial Observability
Peter Karkus1,2 David Hsu1,2 Wee Sun Lee2 1NUS Graduate School for Integrative Sciences and Engineering 2School of Computing National University of Singapore {karkus, dyhsu,
Abstract
This paper introduces the QMDP-net, a neural network architecture for planning under partial observability. The QMDP-net combines the strengths of model-free learning and model-based planning. It is a recurrent policy network, but it represents a policy for a parameterized set of tasks by connecting a model with a planning algorithm that solves the model, thus embedding the solution structure of planning in a network learning architecture. The QMDP-net is fully differentiable and allows for end-to-end training. We train a QMDP-net on different tasks so that it can generalize to new ones in the parameterized task set and “transfer” to other similar tasks beyond the set. In preliminary experiments, QMDP-net showed strong performance on several robotic tasks in simulation. Interestingly, while QMDP-net encodes the QMDP algorithm, it sometimes outperforms the QMDP algorithm in the experiments, as a result of end-to-end learning.
中文速览
在部分可观测环境中做决策是机器人和AI领域的核心难题——智能体无法直接知道自己的真实状态，必须靠历史行动和观测记录来推断。论文提出了一种叫 QMDP-net 的神经网络架构，把贝叶斯滤波器（用于估计状态概率分布）和 QMDP 规划算法（用于根据该分布选择动作）整合成一个完全可微的循环神经网络，从而能够端到端地从专家示范轨迹中训练，同时学习任务的转移、观测和奖励模型。这一设计让网络在一批随机生成的环境上训练后，能够泛化到从未见过的新环境，甚至迁移到更复杂的场景，在多个机器人仿真任务（包括著名的 Hallway2 迷宫导航）上超越了现有的深度学习基线方法。更有趣的是，尽管网络内嵌的是近似的 QMDP 算法，端到端训练使它学会了补偿该算法的局限性，在实验中有时反而比直接运行 QMDP 算法本身表现更好——这说明将规划算法结构嵌入神经网络并配合数据驱动学习，是一条兼顾泛化能力与计算效率的有效路径。
原文 arXiv:1703.06692;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.06692v3