GIT: A Generative Image-to-text Transformer for Vision and Language
Jianfeng Wang Zhengyuan Yang Xiaowei Hu Linjie Li Kevin Lin Zhe Gan Zicheng Liu Ce Liu Lijuan Wang Microsoft Cloud and AI
Abstract
In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on numerous challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks.
中文速览
图文理解任务长期依赖目标检测器、OCR等外部模块以及复杂的编解码结构，微软研究院提出了一个极简的生成式图文转换模型GIT（Generative Image-to-text Transformer），整个架构只用一个图像编码器加一个文本解码器，以语言建模为唯一的预训练目标，彻底去掉了目标检测、OCR等外部依赖。研究团队用8亿图文对大规模预训练，并同步扩大模型体量，让这套"轻装上阵"的设计发挥出最大潜力。最终GIT在图像/视频描述生成、视觉问答等多个主流榜单上刷新了最优成绩，尤其在TextCaps上以138.2分首次超越人类水平（125.5分）。这项工作表明，不需要堆砌复杂模块，靠架构简化加数据与模型规模扩展同样能实现顶级性能，为视觉语言模型的统一建模提供了一条更清晰的路径。
原文 arXiv:2205.14100;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.14100v5