LLM-Based Human-Robot Collaboration Framework for Manipulation Tasks
Haokun Liu1 Yaonan Zhu1∗ Kenji Kato2 Izumi Kondo2 Tadayoshi Aoyama1 and Yasuhisa Hasegawa1 1. Department of Micro-Nano Mechanical Science and Engineering Nagoya University Nagoya Aichi 464-8603 Japan 2. National Center for Geriatrics and Gerontology Obu Aichi 474-8511 Japan ∗Corresponding author email: work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
Abstract
This paper presents a novel approach to enhance autonomous robotic manipulation using the Large Language Model (LLM) for logical inference, converting high-level language commands into sequences of executable motion functions. The proposed system combines the advantage of LLM with YOLO-based environmental perception to enable robots to autonomously make reasonable decisions and task planning based on the given commands. Additionally, to address the potential inaccuracies or illogical actions arising from LLM, a combination of teleoperation and Dynamic Movement Primitives (DMP) is employed for action correction. This integration aims to improve the practicality and generalizability of the LLM-based human-robot collaboration system.
中文速览
让机器人听懂人话并自主完成家务任务，一直面临指令理解不准、动作不可行等挑战。研究者提出了一套以大语言模型（LLM, Large Language Model）为核心的任务规划框架，让GPT-2将用户的自然语言指令转化为可执行的动作函数序列，同时借助YOLO进行实时环境感知来定位目标物体。针对LLM可能产生错误或不合理动作的问题，系统引入了远程操控结合动态运动基元（DMP, Dynamic Movement Primitives）的纠错机制，让人类示范正确轨迹后由机器人复现，从而弥补自动规划的不足。实验表明，机器人在"抓取""放置""打开"等短时任务上取得了较高成功率，而对于"清理柜顶"这类长时任务成功率仍偏低，主要源于动作误差的累积。这项工作的意义在于提供了一种将人类灵活性与机器人自主性有机结合的协作思路，为更实用、更通用的家庭服务机器人系统奠定了基础。
原文 arXiv:2308.14972;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.14972v1