Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Pan Lu1, Baolin Peng2, Hao Cheng2, Michel Galley2 Kai-Wei Chang1, Ying Nian Wu1, Song-Chun Zhu1, Jianfeng Gao2 1University of California, Los Angeles 2Microsoft Research, Redmond https://chameleon-llm.github.io
Abstract
Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by
中文速览
大语言模型（LLM）虽然推理能力出色，却有三个硬伤：不能获取最新信息、无法调用外部工具、难以完成精确的数学与逻辑推理。Chameleon 的思路是用一个基于 LLM 的"规划器"，让它像写程序一样把各种即插即用的工具模块——包括视觉模型、网络搜索、Python 解释器、规则函数等——按需拼接成一条执行流水线来回答复杂问题。在多模态科学问答基准 ScienceQA 上，GPT-4 驱动的 Chameleon 达到 86.54% 的准确率，比此前最佳小样本结果提升了 11.37%；在需要表格数学推理的 TabMWP 上，准确率更是跃升至 98.78%，比最强基线高出 17%。这项工作的价值在于它无需额外训练，只靠自然语言指令就能灵活组合异构工具，为弥补 LLM 自身短板、构建通用 AI 系统提供了一条切实可行的路径。
原文 arXiv:2304.09842;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.09842v3