Learning Generalizable Robotic Reward Functions from “In-The-Wild” Human Videos
Annie S. Chen, Suraj Nair, Chelsea Finn Stanford University
Abstract
We are motivated by the goal of generalist robots that can complete a wide range of tasks across many environments. Critical to this is the robot’s ability to acquire some metric of task success or reward, which is necessary for reinforcement learning, planning, or knowing when to ask for help. For a general-purpose robot operating in the real world, this reward function must also be able to generalize broadly across environments, tasks, and objects, while depending only on on-board sensor observations (e.g. RGB images). While deep learning on large and diverse datasets has shown promise as a path towards such generalization in computer vision and natural language, collecting high quality datasets of robotic interaction at scale remains an open challenge. In contrast, “in-the-wild” videos of humans (e.g. YouTube) contain an extensive collection of people doing interesting tasks across a diverse range of settings. In this work, we propose a simple approach, Domain-agnostic Video Discriminator (DVD), that learns multitask reward functions by training a discriminator to classify whether two videos are performing the same task, and can generalize by virtue of learning from a small amou
中文速览
机器人要在真实世界里自主完成各种任务，就必须有一个能判断"任务做成功了没"的奖励函数，但让它既能跨环境、跨任务泛化，又只靠摄像头图像来工作，一直是个难题。研究者提出了一种叫"领域无关视频判别器"（Domain-agnostic Video Discriminator，DVD）的方法：训练一个分类器来判断两段视频是否在完成同一个任务，同时混合使用少量机器人操作视频和大量来自YouTube等平台的人类日常操作视频（如Something-Something数据集）进行训练，让模型学会跨视觉域的"功能相似性"。实验表明，引入多样化人类视频后，这个奖励函数能零样本泛化到未见过的环境和未见过的任务，比仅用机器人数据的基线方法任务成功率高出15–20个百分点，并最终在真实WidowX200机械臂上，凭一段人类示范视频就在全新环境中完成了操作任务。这项工作的意义在于，它提供了一条绕开大规模机器人数据采集难题的捷径——直接"借用"互联网上已有的海量人类视频来赋予机器人泛化能力。
原文 arXiv:2103.16817;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.16817v1