Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models
Ted Xiao1,*1{}^{1,*}start_FLOATSUPERSCRIPT 1 , * end_FLOATSUPERSCRIPT Harris Chan1,2,*12{}^{1,2,*}start_FLOATSUPERSCRIPT 1 , 2 , * end_FLOATSUPERSCRIPT Pierre Sermanet11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Ayzaan Wahid11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Anthony Brohan11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Karol Hausman11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Sergey Levine11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Jonathan Tompson11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT *{}^{*}start_FLOATSUPERSCRIPT * end_FLOATSUPERSCRIPT Equal contribution 11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPTRobotics at Google 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTUniversity of Toronto Project website: https://instructionaugmentation.github.io
Abstract
Robotic manipulation policies that follow natural language instructions are typically trained from corpora of robot-language data that were either collected with specific tasks in mind or expensively relabeled by humans with varied language descriptions in hindsight. Recently, large-scale pretrained vision-language models (VLMs) have been applied to robotics for learning representations and scene descriptors. Can these pretrained models serve as automatic labelers for robot data, effectively importing Internet-scale knowledge into existing datasets with limited ground truth annotations? For example, if the original annotations contained templated task descriptions such as “pick apple”, a pretrained VLM-based labeler could significantly expand the number of semantic concepts available in the data and introduce spatial concepts such as “the apple on the right side of the table” or alternative phrasings such as “the red colored fruit”. To accomplish this, we introduce Data-driven Instruction Augmentation for Language-conditioned control (DIAL): we utilize semi-supervised language labels to propagate CLIP’s semantic knowledge onto large datasets of unlabeled demonstration data, from wh
中文速览
让机器人听懂更多自然语言指令，关键瓶颈在于：现有的机器人演示数据集虽然规模庞大，但绝大多数演示缺乏语言标注，而人工标注既费时又昂贵。本文提出 DIAL（Data-driven Instruction Augmentation for Language-conditioned Control），核心思路是先用少量人工标注数据对 CLIP 这类视觉-语言大模型（VLM）进行微调，再让微调后的模型自动为海量未标注演示匹配合适的语言指令，从而低成本地把互联网级别的语义知识"注入"到机器人数据集中。在一个包含 8 万条真实机器人演示、但仅有 3.5% 带人工语言标注的厨房操作任务上，研究者进行了超过 1300 次真实机器人评测，结果显示 DIAL 让模仿学习策略在 60 条原数据集中从未出现过的新指令上性能提升超过 41%。这项工作表明，借助预训练 VLM 的自动重标注，可以用极低的额外人力成本大幅拓展机器人的语言理解与泛化能力，对构建可扩展的语言条件机器人控制系统具有重要意义。
原文 arXiv:2211.11736;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.11736v3