Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
Dhruv Shahγβ, Peng Xuγ, Yao Luγ, Ted Xiaoγ Alexander Toshevγ, Sergey Levineγβ, Brian Ichterγ γGoogle Research, Robotics @ Google βBerkeley AI Research, UC Berkeley
Abstract
Reinforcement learning can train policies that effectively perform complex tasks. However for long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and chaining lower-level skills. Hierarchical reinforcement learning aims to enable this by providing a bank of low-level skills as action abstractions. Hierarchies can further improve on this by abstracting the space states as well. We posit that a suitable state abstraction should depend on the capabilities of the available lower-level policies. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill. These value functions capture the affordances of the scene, thus forming a representation that compactly abstracts task relevant information and robustly ignores distractors. Empirical evaluations for maze-solving and robotic manipulation tasks demonstrate that our approach improves long-horizon performance and enables better zero-shot generalization than alternative model-free and model-based methods.
中文速览
长期任务中的机器人或智能体往往需要把多个底层技能（如抓取、放置、开抽屉）串联起来才能完成目标，但如何为上层决策提供一个简洁又有效的状态表示一直是难题。这项工作提出了"价值函数空间"（Value Function Space，VFS）方法：把每个底层技能对应的价值函数（value function）拼接在一起，作为上层策略的状态表示，因为价值函数天然描述了"在当前状态下执行某技能能有多大收益"，即场景对各技能的可供性（affordance）。这样构造出的表示既压缩了高维观测中的冗余信息，又自动忽略与任务无关的干扰因素（如背景纹理、无关物体），让功能等价的场景映射到同一个嵌入向量。在迷宫导航和机器人操作任务上的实验表明，VFS在长时序任务中的表现优于对比学习和基于模型等多种基线方法，还能零样本（zero-shot）泛化到训练中未见过的新环境。这项工作的意义在于，它提供了一种无需额外监督、直接利用已有技能知识来构建规划表示的简单方案，为分层强化学习中的感知抽象问题给出了一个原则性的解答。
原文 arXiv:2111.03189;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.03189v2