Variational Bayesian Reinforcement Learning with Regret Bounds
Brendan O’Donoghue DeepMind, UK
Abstract
In reinforcement learning the Q-values summarize the expected future rewards that the agent will attain. However, they cannot capture the epistemic uncertainty about those rewards. In this work we derive a new Bellman operator with associated fixed point we call the ‘knowledge values’. These K-values compress both the expected future rewards and the epistemic uncertainty into a single value, so that high uncertainty, high reward, or both, can yield high K-values. The key principle is to endow the agent with a risk-seeking utility function that is carefully tuned to balance exploration and exploitation. When the agent follows a Boltzmann policy over the K-values it yields a Bayes regret bound of $\tilde{O}(L\sqrt{SAT})$ , where $L$ is the time horizon, $S$ is the total number of states, $A$ is the number of actions, and $T$ is the number of elapsed timesteps. We show deep connections of this approach to the soft-max and maximum-entropy strands of research in reinforcement learning.
中文速览
强化学习中的Q值只能描述预期未来奖励，却无法衡量智能体对环境的认知不确定性，导致探索与利用的权衡难以兼顾。这项工作提出了一种新的Bellman算子，其不动点被称为"知识值"（K-values），通过让智能体具备一种风险偏好的指数效用函数，将预期奖励与认知不确定性压缩进同一个标量——奖励高、不确定性高或二者兼备，都会产生高K值。基于K值采用Boltzmann策略的K-learning算法，被证明可以达到$\tilde{O}(L\sqrt{SAT})$的贝叶斯遗憾上界，与Thompson采样的最优已知界仅差对数因子。这一工作首次从原则性贝叶斯视角证明了软最大（soft-max）算子、最大熵策略和Boltzmann探索能够获得可证明的良好性能，为这类广泛使用的技术提供了坚实的理论基础。
原文 arXiv:1807.09647;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1807.09647v4