Maximum Entropy Deep Inverse Reinforcement Learning
Markus Wulfmeier Peter Ondrúška Ingmar Posner Mobile Robotics Group, Department of Engineering Science, University of Oxford
Abstract
This paper presents a general framework for exploiting the representational capacity of neural networks to approximate complex, nonlinear reward functions in the context of solving the inverse reinforcement learning (IRL) problem. We show in this context that the Maximum Entropy paradigm for IRL lends itself naturally to the efficient training of deep architectures. At test time, the approach leads to a computational complexity independent of the number of demonstrations, which makes it especially well-suited for applications in life-long learning scenarios. Our approach achieves performance commensurate to the state-of-the-art on existing benchmarks while exceeding on an alternative benchmark based on highly varying reward structures.Finally, we extend the basic architecture - which is equivalent to a simplified subclass of Fully Convolutional Neural Networks (FCNNs) with width one - to include larger convolutions in order to eliminate dependency on precomputed spatial features and work on raw input representations.
中文速览
逆强化学习（Inverse Reinforcement Learning, IRL）的核心难题在于：如何从专家示范中反推出驱动行为的奖励函数，尤其是当奖励结构非常复杂、非线性时，现有方法要么表达能力不足，要么计算开销随示范数量急剧增加。这篇论文提出了一个基于最大熵（Maximum Entropy）范式的深度逆强化学习框架（Deep IRL），用深度神经网络来近似奖励函数，使目标函数对网络权重完全可微，从而可以直接用反向传播训练。关键优势在于，神经网络在查询时的计算复杂度与示范数量无关（O(1)），天然适合终身学习等需要持续积累经验的场景；进一步将网络扩展为全卷积结构（Fully Convolutional Neural Network, FCNN）后，模型还能直接从原始输入中学习空间特征，无需人工设计特征。实验表明，该方法在已有基准上达到业界领先水平，在奖励结构高度复杂的新基准上更是超越了现有方法，为大规模、复杂场景下的 IRL 应用提供了兼具表达能力与计算效率的解决方案。
原文 arXiv:1507.04888;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1507.04888v3