When Waiting is not an Option : Learning Options with a Deliberation Cost
Jean Harb∗, Pierre-Luc Bacon, Martin Klissarov, Doina Precup Reasoning and Learning Lab, McGill University Indicates equal contribution.
Abstract
Recent work has shown that temporally extended actions (options) can be learned fully end-to-end as opposed to being specified in advance. While the problem of how to learn options is increasingly well understood, the question of what good options should be has remained elusive. We formulate our answer to what good options should be in the bounded rationality framework (Simon, 1957) through the notion of deliberation cost. We then derive practical gradient-based learning algorithms to implement this objective. Our results in the Arcade Learning Environment (ALE) show increased performance and interpretability.
中文速览
强化学习中的智能体在选择动作时面临一个根本问题：每次都从头决策既耗时又低效，而"时序抽象动作"（options，即可以持续执行多步的宏观动作）本该帮助智能体节省决策开销，但究竟什么样的options才是"好的options"，一直缺乏清晰的理论依据。这篇论文借用有界理性（bounded rationality）框架，提出了"审议成本"（deliberation cost）这一概念来回答这个问题——每次切换到新option都要付出一定的计算代价，而在option内部继续执行则是免费的，因此更长、更稳定的options能够更好地分摊切换成本，从而让智能体学得更快、规划更高效。基于这一思想，作者推导出了可以用梯度下降直接优化的学习算法，并将其嵌入option-critic框架中在Atari游戏环境（Arcade Learning Environment）上进行验证，结果显示智能体不仅获得了更高的游戏得分，学到的options也更具可解释性。这项工作的意义在于，它首次为"好的时序抽象动作"提供了一个有理论支撑的形式化目标，将原本模糊的设计直觉转化为可优化的数学准则。
原文 arXiv:1709.04571;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.04571v1