VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
Wenlong Huang1, Chen Wang1, Ruohan Zhang1, Yunzhu Li1,2, Jiajun Wu1, Li Fei-Fei1 1Stanford University 2University of Illinois Urbana-Champaign
Abstract
Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes tha
中文速览
大型语言模型（LLM）其实已经储存了大量关于物理世界的常识，但过去的机器人系统想用上这些知识时，仍然离不开人工预设的动作原语（motion primitives），严重制约了机器人能完成的任务种类。VoxPoser 的核心思路是：让 LLM 自动写 Python 代码，调用视觉语言模型（VLM）感知场景中物体的三维位置，再用 NumPy 把"应该去哪里、应该躲开什么"编码成一张稠密的三维体素价值图（3D voxel value map），然后把这张图当作目标函数，直接驱动基于模型的运动规划器实时生成机器人末端执行器的六自由度轨迹。实验结果表明，这套方法无需针对特定任务收集训练数据，就能零样本泛化到大量日常操作任务，在仿真和真实机器人上均表现出色，还能通过少量在线交互高效学习接触丰富场景的动力学模型。这项工作的重要意义在于，它填补了 LLM 高层推理与机器人底层物理执行之间长期存在的鸿沟，为开放指令、开放物体场景下的通用机器人操作提供了一条无需大规模机器人数据的新路径。
原文 arXiv:2307.05973;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2307.05973v2