R3M: A Universal Visual Representation for Robot Manipulation
Suraj Nair1,∗, Aravind Rajeswaran2, Vikash Kumar2, Chelsea Finn1, Abhinav Gupta2 1Stanford University, 2Meta AI
Abstract
We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks. Concretely, we pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations. The resulting representation, R3M, can be used as a frozen perception module for downstream policy learning. Across a suite of 12 simulated robot manipulation tasks, we find that R3M improves task success by over $20\%$ compared to training from scratch and by over $10\%$ compared to state-of-the-art visual representations like CLIP and MoCo. Furthermore, R3M enables a Franka Emika Panda arm to learn a range of manipulation tasks in a real, cluttered apartment given just 20 demonstrations. Code and pre-trained models are available at https://tinyurl.com/robotr3m.
中文速览
机器人操控任务通常需要海量带标注的机器人数据从头训练，既昂贵又难以泛化；R3M（Reusable Representations for Robotic Manipulation）另辟蹊径，利用人类第一视角视频数据集 Ego4D（超过 3500 小时的日常交互视频）来预训练一个通用视觉表征模型。具体做法是联合使用时序对比学习（time-contrastive learning）捕捉动态信息、视频-语言对齐（video-language alignment）提取语义特征，以及 L1 稀疏正则化使表征更紧凑，训练完成后将该编码器"冻结"直接用于下游策略学习。在 12 个仿真机器人操控任务上，R3M 比从头训练提升超过 20%，比 CLIP、MoCo 等主流视觉表征提升超过 10%；在真实杂乱公寓场景中，仅凭不到 20 条示范，Franka 机械臂即可以 50% 以上的成功率完成放生菜入锅、折叠毛巾等复杂任务，约是 CLIP 特征成功率的两倍。这项工作表明，多样化的人类视频完全可以替代昂贵的机器人数据，为构建像 ImageNet/BERT 那样"下载即用"的机器人通用视觉基础模型提供了切实可行的路径。
原文 arXiv:2203.12601;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.12601v3