An Interactive Agent Foundation Model
Zane Durante Affiliation: Stanford University Affiliation: Microsoft Research, Redmond Bidipta Sarkar Affiliation: Stanford University Affiliation: Microsoft Research, Redmond Ran Gong Affiliation: University of California, Los Angeles Affiliation: Microsoft Research, Redmond Rohan Taori Affiliation: Stanford University Affiliation: Microsoft Research, Redmond Yusuke Noda Affiliation: Microsoft Research, Redmond Paul Tang Affiliation: Stanford University Ehsan Adeli Affiliation: Stanford University Shrinidhi Kowshika Lakshmikanth Affiliation: Stanford University Kevin Schulman Affiliation: Stanford University Arnold Milstein Affiliation: Stanford University Demetri Terzopoulos Affiliation: University of California, Los Angeles Ade Famoti Affiliation: Microsoft Research, Redmond Noboru Kuno Affiliation: Microsoft Research, Redmond Ashley Llorens Affiliation: Microsoft Research, Redmond Hoi Vo Affiliation: Microsoft Research, Redmond Katsu Ikeuchi Affiliation: Microsoft Research, Redmond Li Fei-Fei Affiliation: Stanford University Jianfeng Gao Affiliation: Microsoft Research, Redmond Naoki Wake Affiliation: Microsoft Research, Redmond Qiuyuan Huang Affiliation: Microsoft Research, Redmond
Abstract
The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction, enabling a versatile and adaptable AI framework. We demonstrate the performance of our framework across three separate domains—Robotics, Gaming AI, and Healthcare. Our model demonstrates its ability to generate meaningful and contextually relevant outputs in each area. The strength of our approach lies in its generality, leveraging a variety of data sources such as robotics sequences, gameplay data, large-scale video datasets, and textual information for effective multimodal and multi-task learning. Our approach provides a promising avenue for developing generalist, action-taking, multimodal systems.
原文 arXiv:2402.05929;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2402.05929v2