“No, to the Right” – Online Language Corrections for Robotic Manipulation via Shared Autonomy
Yuchen Cui Stanford UniversityStanfordCAUSA , Siddharth Karamcheti Stanford UniversityStanfordCAUSA , Raj Palleti Stanford UniversityStanfordCAUSA , Nidhya Shivakumar The Harker SchoolSan JoseCAUSA , Percy Liang Stanford UniversityStanfordCAUSA and Dorsa Sadigh Stanford UniversityStanfordCAUSA
Abstract
Systems for language-guided human-robot interaction must satisfy two key desiderata for broad adoption: adaptivity and learning efficiency. Unfortunately, existing instruction-following agents cannot adapt, lacking the ability to incorporate online natural language supervision, and even if they could, require hundreds of demonstrations to learn even simple policies. In this work, we address these problems by presenting Language-Informed Latent Actions with Corrections (LILAC), a framework for incorporating and adapting to natural language corrections – “to the right”, or “no, towards the book” – online, during execution. We explore rich manipulation domains within a shared autonomy paradigm. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot: language is an input to a learned model that produces a meaningful, low-dimensional control space that the human can use to guide the robot. Each real-time correction refines the human’s control space, enabling precise, extended behaviors – with the added benefit of requiring only a handful of demonstrations to learn. We evaluate our approach via a user study where users work with a Frank
中文速览
让机器人听懂人类实时口头纠正指令、并据此调整动作，是人机协作中一个悬而未决的难题：现有系统要么只能"说一次、跑到底"，无法在执行过程中响应新指令，要么需要海量示范数据才能学会简单任务。LILAC（语言引导的潜在动作纠正框架）把语言指令引入"共享自主"范式——系统用少量示范学出一个低维控制空间，人类用摇杆在这个空间里导引机器人，每当用户说出"往右一点"或"朝那本书的方向"这类实时纠正语句，控制空间就随之在线更新，让用户能精确完成插书入架这类需要毫米级对齐的复杂操作。在真实Franka机械臂上进行的12人用户研究表明，LILAC的任务完成率显著高于两条已有基线（开环指令跟随和单轮共享自主），参与者也普遍认为它更可靠、更精准、更易用。这项工作的意义在于，它仅用10–20条示范就能支撑复杂操作任务，同时提供了一条让机器人真正"边干边听纠正"的实用路径。
原文 arXiv:2301.02555;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.02555v1