ViperGPT: Composing Vision and Language via Code
Dídac Surís , Sachit Menon*, Carl Vondrick Columbia University viper.cs.columbia.edu Equal contribution. Order determined via coin flip and may be listed either way.
Abstract
Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ViperGPT utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.
中文速览
回答视觉问题既要"看懂图"，又要"想明白"，现有端到端神经网络把这两件事混在一个黑箱里完成，既难解释又难推广到新任务。ViperGPT 的做法是：把各种现成的视觉和语言模型（如目标检测、深度估计、视觉问答）封装成一套 Python API，然后让代码生成大语言模型 Codex 根据用户的文字问题自动写出一段 Python 程序，再由解释器执行这段程序得到答案——整个过程无需针对具体任务重新训练任何模型。在视觉定位、组合式图像问答、依赖外部知识的问答以及视频时序推理等多个基准测试上，ViperGPT 取得了零样本（zero-shot）最优结果，同时生成的代码本身就是逐步可查的推理过程，兼顾了性能与可解释性。这项工作表明，用代码来"编排"已有视觉模型是一条无需大规模重训练、却能解决复杂视觉推理任务的可行路径。
原文 arXiv:2303.08128;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.08128v1