An Interactive Agent Foundation Model
Zane Durante Bidipta Sarkar Ran Gong Rohan Taori Yusuke Noda Paul Tang Ehsan Adeli Shrinidhi Kowshika Lakshmikanth Kevin Schulman Arnold Milstein Demetri Terzopoulos Ade Famoti Noboru Kuno Ashley Llorens Hoi Vo Katsu Ikeuchi Li Fei-Fei Jianfeng Gao Naoki Wake Qiuyuan Huang
Abstract
The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction, enabling a versatile and adaptable AI framework. We demonstrate the performance of our framework across three separate domains—Robotics, Gaming AI, and Healthcare. Our model demonstrates its ability to generate meaningful and contextually relevant outputs in each area. The strength of our approach lies in its generality, leveraging a variety of data sources such as robotics sequences, gameplay data, large-scale video datasets, and textual information for effective multimodal and multi-task learning. Our approach provides a promising avenue for developing generalist, action-taking, multimodal systems.
中文速览
如今AI领域正在从"只会完成单一任务的静态模型"向"能在各种场景中主动行动的智能体"转型，但现有的大模型往往因为缺乏与真实或虚拟环境的深度绑定而频繁"幻觉"出错。研究者提出了一种"交互式智能体基础模型"（Interactive Agent Foundation Model），将视觉、语言和动作三类信号统一为token序列，同时用遮蔽图像自编码、语言建模和下一步动作预测三种目标联合预训练一个2.77亿参数的模型，训练数据横跨机器人操作、游戏AI和医疗健康三个领域共1340万帧视频。实验表明，这一单一模型能在上述三个截然不同的领域中有效完成视觉问答、场景描述、具身导航与动作预测等多种任务，展现出良好的跨域泛化能力。这项工作的意义在于它提供了一条将感知、规划与人机交互融为一体的通用智能体训练路径，为迈向真正"会行动"的多模态通用AI奠定了基础。
原文 arXiv:2402.05929;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2402.05929v2