AppAgent: Multimodal Agents as Smartphone Users
Chi Zhang∗ Zhao Yang∗ Jiaxuan Liu∗ Yucheng Han Xin Chen Zebiao Huang Bin Fu Gang Yu† Tencent {johnczhang, jayzyang, jiaxuanliu, yuchenghan, shingxchen, zebiaohuang, brianfu, https://appagent-official.github.io/
Abstract
Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent’s functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent’s proficiency in handling a diverse array of high-level tasks.
中文速览
智能手机上充斥着各式各样的App，如何让AI像真人一样通过点击、滑动界面来操控这些App，而无需接触系统后台，是一个极具挑战性的问题。研究团队提出了一套基于多模态大语言模型（multimodal LLM）的智能体框架：agent先通过自主探索或观察人类演示来"学习"一款App的界面逻辑，将所学整理成知识文档，之后在执行任务时随时查阅该文档来决定下一步操作，整个过程无需对模型进行额外训练或收集大规模标注数据。在涵盖社交媒体、邮件、地图、购物和图像编辑等10款App、共50个任务的测试中，该框架展现出较强的任务完成能力，用户研究也证实了其易用性与灵活性。这项工作的意义在于，它为AI全面接管手机操作提供了一条无需系统权限、跨App通用的可行路径，大幅降低了智能手机自动化的门槛。
原文 arXiv:2312.13771;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2312.13771v3