Aha.
正在载入中英对照阅读…

arXiv:2606.19120 · 中英对照阅读

先看再推理:解耦感知与推理,实现抗捷径的多模态在策略自蒸馏

Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation

Sihan Wang、Xiyao Liu、Lianqing Liu、Zhi Han

中文速览

多模态模型做自我蒸馏时,可能会更依赖参考答案而不是图片,学会“答案对了、看图不准”的捷径。为此,ViGOS让模型先描述图片,再进行推理和作答:描述阶段由只能看图的教师指导,推理和答案阶段才由能看到参考答案的教师指导,格式错误时另用教师帮忙纠正格式。实验显示,这种方法保留了自我蒸馏在常规基准上的优势,也提升了模型在容易受语言偏见影响的任务中依据图像作答的能力。它的重要性在于把“看清图片”和“根据证据推理”分开训练,减少模型被答案带偏的风险。

摘要

在策略自蒸馏(OPSD)通过模型自身的采样轨迹训练模型,并使用冻结副本提供以参考目标为条件的密集词元级目标。这种方法在大语言模型(LLM)推理任务中效果良好,但将其直接扩展到多模态大语言模型(MLLM)可能会产生捷径:特权目标可能主要依据文本参考目标而非图像来引导词元。我们提出 ViGOS,一种用于多模态大语言模型后训练的视觉锚定在策略自蒸馏框架。学生首先撰写视觉描述,然后推理得出最终答案。对于有效的采样轨迹,图像专属感知教师负责监督描述,而特权推理教师则在相同的学生前缀上监督推理过程和最终答案。参考教师仅用于无效采样轨迹,以恢复输出格式。在通用视觉语言、专家推理、视觉数学、空间锚定和视觉语言先验基准测试中,ViGOS 保留了在策略自蒸馏的主要优势,并提升了容易受到捷径影响的场景中的图像锚定行为。

On-policy self-distillation (OPSD) trains a model on its own rollouts and uses a frozen copy to provide dense token-level targets conditioned on a reference target. This works well for LLM reasoning, but a direct extension to multimodal large language models (MLLMs) can create a shortcut: the privileged target may guide tokens mainly based on the text reference target rather than the image. We propose ViGOS, a visually grounded OPSD framework for MLLM post-training. The student first writes a visual description and then reasons toward the final answer. For valid rollouts, an image-only perception teacher supervises the description, while a privileged reasoning teacher supervises the reasoning and final answer on the same student prefix. A reference teacher is used only for invalid rollouts to recover the output format. Across general vision-language, expert reasoning, visual math, spatial grounding, and visual-language-prior benchmarks, ViGOS keeps the main benefits of OPSD and improves image-grounded behavior in shortcut-prone settings.

术语表

On-policy distillation (OPD)
在策略蒸馏
On-policy self-distillation (OPSD)
在策略自蒸馏
Multimodal large language model (MLLM)
多模态大语言模型
Visual Grounding On-Policy Self-Distillation (ViGOS)
视觉锚定在策略自蒸馏
post-training
后训练
rollout
采样轨迹
student policy
学生策略
frozen teacher
冻结教师
token-level supervision
词元级监督
dense feedback
密集反馈
reinforcement learning with verifiable rewards (RLVR)
可验证奖励强化学习
multimodal reasoning
多模态推理
visual grounding
视觉锚定
visual description
视觉描述
image-only perception teacher
仅图像感知教师
privileged reasoning teacher
特权推理教师
reference teacher
参考教师
language prior
语言先验
shortcut learning
捷径学习
visual question answering (VQA)
视觉问答
visual mathematics
视觉数学
spatial grounding
空间锚定
Privileged Answer Leakage Rate (PALR)
特权答案泄漏率
counterfactual
反事实
teacher distribution
教师分布
student prefix
学生前缀
answer-conditioned supervision
答案条件监督
multimodal shortcut
多模态捷径