arXiv:2606.19120 · 中英对照阅读
先看再推理:解耦感知与推理,实现抗捷径的多模态在策略自蒸馏
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
中文速览
多模态模型做自我蒸馏时,可能会更依赖参考答案而不是图片,学会“答案对了、看图不准”的捷径。为此,ViGOS让模型先描述图片,再进行推理和作答:描述阶段由只能看图的教师指导,推理和答案阶段才由能看到参考答案的教师指导,格式错误时另用教师帮忙纠正格式。实验显示,这种方法保留了自我蒸馏在常规基准上的优势,也提升了模型在容易受语言偏见影响的任务中依据图像作答的能力。它的重要性在于把“看清图片”和“根据证据推理”分开训练,减少模型被答案带偏的风险。
摘要
在策略自蒸馏(OPSD)通过模型自身的采样轨迹训练模型,并使用冻结副本提供以参考目标为条件的密集词元级目标。这种方法在大语言模型(LLM)推理任务中效果良好,但将其直接扩展到多模态大语言模型(MLLM)可能会产生捷径:特权目标可能主要依据文本参考目标而非图像来引导词元。我们提出 ViGOS,一种用于多模态大语言模型后训练的视觉锚定在策略自蒸馏框架。学生首先撰写视觉描述,然后推理得出最终答案。对于有效的采样轨迹,图像专属感知教师负责监督描述,而特权推理教师则在相同的学生前缀上监督推理过程和最终答案。参考教师仅用于无效采样轨迹,以恢复输出格式。在通用视觉语言、专家推理、视觉数学、空间锚定和视觉语言先验基准测试中,ViGOS 保留了在策略自蒸馏的主要优势,并提升了容易受到捷径影响的场景中的图像锚定行为。
On-policy self-distillation (OPSD) trains a model on its own rollouts and uses a frozen copy to provide dense token-level targets conditioned on a reference target. This works well for LLM reasoning, but a direct extension to multimodal large language models (MLLMs) can create a shortcut: the privileged target may guide tokens mainly based on the text reference target rather than the image. We propose ViGOS, a visually grounded OPSD framework for MLLM post-training. The student first writes a visual description and then reasons toward the final answer. For valid rollouts, an image-only perception teacher supervises the description, while a privileged reasoning teacher supervises the reasoning and final answer on the same student prefix. A reference teacher is used only for invalid rollouts to recover the output format. Across general vision-language, expert reasoning, visual math, spatial grounding, and visual-language-prior benchmarks, ViGOS keeps the main benefits of OPSD and improves image-grounded behavior in shortcut-prone settings.
术语表
- On-policy distillation (OPD)
- 在策略蒸馏
- On-policy self-distillation (OPSD)
- 在策略自蒸馏
- Multimodal large language model (MLLM)
- 多模态大语言模型
- Visual Grounding On-Policy Self-Distillation (ViGOS)
- 视觉锚定在策略自蒸馏
- post-training
- 后训练
- rollout
- 采样轨迹
- student policy
- 学生策略
- frozen teacher
- 冻结教师
- token-level supervision
- 词元级监督
- dense feedback
- 密集反馈
- reinforcement learning with verifiable rewards (RLVR)
- 可验证奖励强化学习
- multimodal reasoning
- 多模态推理
- visual grounding
- 视觉锚定
- visual description
- 视觉描述
- image-only perception teacher
- 仅图像感知教师
- privileged reasoning teacher
- 特权推理教师
- reference teacher
- 参考教师
- language prior
- 语言先验
- shortcut learning
- 捷径学习
- visual question answering (VQA)
- 视觉问答
- visual mathematics
- 视觉数学
- spatial grounding
- 空间锚定
- Privileged Answer Leakage Rate (PALR)
- 特权答案泄漏率
- counterfactual
- 反事实
- teacher distribution
- 教师分布
- student prefix
- 学生前缀
- answer-conditioned supervision
- 答案条件监督
- multimodal shortcut
- 多模态捷径