Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei† Microsoft https://github.com/microsoft/unilm Equal contribution. ††\dagger Corresponding author.
Abstract
A big convergence of language, multimodal perception, action, and world modeling is a key step toward artificial general intelligence. In this work, we introduce Kosmos-1111Kosmos is pronounced as and means “Cosmos”., a Multimodal Large Language Model (MLLM) that can perceive general modalities, learn in context (i.e., few-shot), and follow instructions (i.e., zero-shot). Specifically, we train Kosmos-1 from scratch on web-scale multimodal corpora, including arbitrarily interleaved text and images, image-caption pairs, and text data. We evaluate various settings, including zero-shot, few-shot, and multimodal chain-of-thought prompting, on a wide range of tasks without any gradient updates or finetuning. Experimental results show that Kosmos-1 achieves impressive performance on (i) language understanding, generation, and even OCR-free NLP (directly fed with document images), (ii) perception-language tasks, including multimodal dialogue, image captioning, visual question answering, and (iii) vision tasks, such as image recognition with descriptions (specifying classification via text instructions). We also show that MLLMs can benefit from cross-modal transfer, i.e., transfer knowledg
中文速览
大语言模型只能处理文字，而真正的通用智能需要同时理解图像、语言等多种模态信息——Kosmos-1正是为填补这一鸿沟而生。研究团队从头开始在海量网络多模态数据上训练这个多模态大语言模型（Multimodal Large Language Model, MLLM），训练数据涵盖纯文本、图文交错文档以及图像-描述配对，并借助冻结的CLIP视觉编码器和一个注意力池化模块将图像信息对齐到语言模型空间。实验表明，Kosmos-1无需微调即可在语言理解、图像描述、视觉问答、OCR、零样本图像分类以及非语言推理（IQ测试）等多类任务上取得令人印象深刻的成绩，还展示了跨模态知识迁移的潜力——多模态训练能反过来提升纯语言的常识推理能力。这项工作为把语言、感知、行动与世界建模统一进一个通用接口提供了有力示范，是迈向通用人工智能的重要一步。
原文 arXiv:2302.14045;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2302.14045v2