\NAME: A Jointly-Scaled Multilingual Language-Image Model
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut Google Research Correspondence:
Abstract
Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present \NAME (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. \NAMEgenerates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train \NAME, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train \NAME, we create a large multilingual mix of pre-training tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. \NAMEachieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text unders
中文速览
如何让视觉-语言模型同时做好图文理解、图像描述、多语言问答等一大堆任务，是当前多模态大模型面临的核心挑战。谷歌这篇论文提出了 PaLI（Pathways Language and Image model），通过将预训练好的超大语言模型 mT5-XXL（130亿参数）和新训练的超大视觉模型 ViT-e（40亿参数）拼接为一个统一的"图文输入→文本输出"框架，并在包含100多种语言、100亿张图片的 WebLI 数据集上进行多任务预训练。关键发现是：视觉部分的扩容和语言部分扩容同样重要，甚至性价比更高，两者齐头并进才能最大化性能，而不是像以往模型那样把绝大多数参数堆在语言侧。最终 PaLI-17B 在 COCO 图像描述（CIDEr 149.1）、VQAv2（84.3%）等多项权威基准上刷新了当时的最优成绩，还能流畅处理百余种语言，为未来多模态大模型的规模化路径提供了清晰的参考蓝图。
原文 arXiv:2209.06794;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2209.06794v4