Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Jinze Bai∗ Shuai Bai∗ Shusheng Yang∗ Shijie Wang Sinan Tan Peng Wang Junyang Lin Chang Zhou† Jingren Zhou Alibaba Group Code、Demo、Models: https://github.com/QwenLM/Qwen-VL
Abstract
In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline, and (iv) multilingual multimodal cleaned corpus. Beyond the conventional image description and question-answering, we implement the grounding and text-reading ability of Qwen-VLs by aligning image-caption-box tuples. The resulting models, including Qwen-VL and Qwen-VL-Chat, set new records for generalist models under similar model scales on a broad range of visual-centric benchmarks (e.g., image captioning, question answering, visual grounding) and different settings (e.g., zero-shot, few-shot). Moreover, on real-world dialog benchmarks, our instruction-tuned Qwen-VL-Chat also demonstrates superiority compared to existing vision-language chatbots. All models are public to facilitate future research.
中文速览
阿里开源的Qwen-VL系列模型要解决的核心问题是：现有开源大型视觉语言模型（Large Vision-Language Model, LVLM）能力偏弱，尤其在细粒度视觉理解（如目标定位和文字识别）上严重不足。研究团队以Qwen-7B语言模型为底座，引入视觉编码器和位置感知适配器，并通过"大规模弱监督预训练→多任务精细化预训练→指令微调"三阶段流程，在14亿清洗后的图文对及多种高质量标注数据上完成训练，同时以图像-描述-坐标框三元组对齐的方式赋予模型目标定位与OCR能力。最终得到的Qwen-VL（基础版）和Qwen-VL-Chat（对话版）在图像描述、视觉问答、视觉定位等主流基准上均刷新了同量级开源模型的最佳成绩，对话版还在真实交互评测中超越了现有视觉语言聊天机器人。这项工作的意义在于向开源社区提供了一个同时具备中英文、多图理解和细粒度感知能力的强基线模型，有助于推动后续研究与落地应用。
原文 arXiv:2308.12966;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.12966v3