Visual Programming: Compositional visual reasoning without training
Tanmay Gupta, Aniruddha Kembhavi PRIOR @ Allen Institute for AI https://prior.allenai.org/projects/visprog
Abstract
We present VisProg, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VisProg avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing subroutines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VisProg on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like VisProg are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.
中文速览
面对"如何让AI系统无需针对每个复杂视觉任务单独训练就能完成它"这一难题，研究者提出了VisProg——一个让大语言模型GPT-3根据自然语言指令自动生成并执行Python风格视觉程序的神经符号系统。系统的核心思路是：只需给GPT-3提供少量"指令-程序"示范样例，它就能为新指令生成一段调用多个现成视觉模型（如目标检测、CLIP、图像编辑等）和Python工具的模块化程序，再由解释器逐行执行得到最终答案。研究者在四类任务上验证了这一框架：组合式视觉问答（比基线模型提升2.7个百分点）、图像对零样本推理（准确率62.4%）、需要外部知识的目标标注，以及语言引导的图像编辑，均取得了令人满意的结果。更关键的是，程序每一步的中间输出都会被整合成可读的"视觉推理链"，让用户能直接看懂系统在想什么、哪里出了错，从而以极低成本扩展AI的能力边界，为解决现实世界中数量庞大的复杂视觉任务提供了一条无需大量标注数据的新路径。
原文 arXiv:2211.11559;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.11559v1