Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence Google
Abstract
Large pretrained (e.g., “foundation”) models exhibit distinct capabilities depending on the domain of data they are trained on. While these domains are generic, they may only barely overlap. For example, visual-language models (VLMs) are trained on Internet-scale image captions, but large language models (LMs) are further trained on Internet-scale text with no images (e.g., spreadsheets, SAT questions, code). As a result, these models store different forms of commonsense knowledge across different domains. In this work, we show that this diversity is symbiotic, and can be leveraged through Socratic Models (SMs): a modular framework in which multiple pretrained models may be composed zero-shot i.e., via multimodal-informed prompting, to exchange information with each other and capture new multimodal capabilities, without requiring finetuning. With minimal engineering, SMs are not only competitive with state-of-the-art zero-shot image captioning and video-to-text retrieval, but also enable new applications such as (i) answering free-form questions about egocentric video, (ii) engaging in multimodal assistive dialogue with people (e.g., for cooking recipes) by interfacing with externa
中文速览
不同大模型在各自领域积累了不同形式的知识——视觉语言模型（VLM）擅长把文字和图像对应起来，大语言模型（LM）则精通阅读理解、逻辑推理，两者几乎从未被放在一起协同工作。这篇论文提出了"苏格拉底模型"（Socratic Models，SMs）框架：把多个独立的预训练大模型像苏格拉底式对话一样串联起来，通过精心设计的文字提示（prompt）让它们互相"交流"、传递信息，完全不需要额外的微调训练。实验结果表明，这种零样本组合方式在图像描述、视频文本检索等标准测试上达到甚至超过了同类零样本方法的最优水平，还解锁了全新应用场景，包括回答第一人称视频中的开放性问题、与用户进行多模态烹饪助手对话，以及驱动机器人完成感知与规划任务。这项工作的意义在于：它证明了无需昂贵的联合训练和海量标注数据，仅靠创造性的提示工程就能把现有大模型的能力拼接成全新的多模态智能，为资源有限或数据稀缺的场景提供了一条切实可行的新路径。
原文 arXiv:2204.00598;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.00598v2