Frozen Transformers in Language Models Are Effective Visual Encoder Layers
Ziqi Pang Ziyang Xie Yunze Man∗ Yu-Xiong Wang University of Illinois Urbana-Champaign https://github.com/ziqipang/LM4VisualEncoding Equal contribution.
Abstract
This paper reveals that large language models (LLMs), despite being trained solely on text data, are surprisingly strong encoders for purely visual tasks in the absence of language. Even more intriguingly, this can be achieved by a simple yet previously overlooked strategy – employing a frozen transformer block from pre-trained LLMs as a constituent encoder layer to directly process visual tokens. Our work pushes the boundaries of leveraging LLMs for computer vision tasks, significantly departing from conventional practices that typically necessitate a multi-modal vision-language setup with associated language prompts, inputs, or outputs. We demonstrate that our approach consistently enhances performance across a diverse range of tasks, encompassing purely 2D and 3D visual recognition tasks (e.g., image and point cloud classification), temporal modeling tasks (e.g., action recognition), non-semantic tasks (e.g., motion forecasting), and multi-modal tasks (e.g., 2D/3D visual question answering and image-text retrieval). Such improvements are a general phenomenon, applicable to various types of LLMs (e.g., LLaMA and OPT) and different LLM transformer blocks.
中文速览
用文本数据训练出来的大语言模型（LLM），其实也能悄悄胜任纯视觉任务——只需从预训练LLM中取出一个冻结的Transformer块，直接插入视觉编码器的末端，再在前后各加一层可训练的线性映射来对齐维度，整个改动极其简单却被此前研究者忽视。实验覆盖了图像分类、点云分类、动作识别、运动预测、视觉问答和图文检索等多种任务，结果显示这一做法在几乎所有场景下都能稳定提升性能，且对LLaMA、OPT等不同系列的LLM普遍有效。为了解释"为何从未见过图像的语言模型能帮助视觉编码"，作者提出"信息过滤假说"：LLM的Transformer块能够识别出更具信息量的视觉token并放大其在特征表示中的贡献，这一点通过可视化激活图得到了验证。这项发现拓宽了LLM的应用边界，提示我们语言模型所习得的通用序列处理能力可能远超语言本身，对理解LLM内在机制和设计跨模态系统都具有启发意义。
原文 arXiv:2310.12973;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.12973v2