Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Chenfei Wu Shengming Yin Weizhen Qi Xiaodong Wang Zecheng Tang Nan Duan Microsoft Research Asia {chewu, v-sheyin, t-weizhenqi, v-xiaodwang, v-zetang, Corresponding author.
Abstract
ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called Visual ChatGPT, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating
中文速览
ChatGPT虽然对话能力出色，却无法处理图像，而那些擅长看图、画图的视觉基础模型（Visual Foundation Models，VFMs）又只能完成单一固定任务、不支持多轮交互——为了打通这两类系统，研究者构建了一个名为 Visual ChatGPT 的框架，通过设计一个"提示管理器"（Prompt Manager）将22种VFM的功能、输入输出格式统一转化为语言描述注入ChatGPT，让ChatGPT像调度员一样按需串联多个视觉模型、逐步完成复杂的图文任务。用户既可以上传图片并用自然语言下达多步骤指令（比如"根据这张图的深度信息生成一朵红花，再把它变成卡通风格"），也可以对中间结果给出反馈要求修改，系统会自动规划调用链并最终返回满足要求的图像或文字。大量零样本实验验证了该系统在图像理解、生成与编辑上的有效性，这项工作为无需重新训练新模型、就能将任意视觉能力"插件化"接入对话系统提供了一条可扩展的新路径。
原文 arXiv:2303.04671;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.04671v1