arXiv：cs.AI· Jiahe Feng·· 16 小时前
VC-OPD：面向视觉语言推理的视觉反事实在线策略蒸馏
VC-OPD: Visual Counterfactual On-Policy Distillation for Grounded Vision-Language Reasoning
AI 导读
研究者提出 VC-OPD，一种面向视觉语言推理的在线策略蒸馏（OPD）方法，通过对比原始图像与损坏实例证据的反事实图像下的教师分布差异，估计 token 级视觉依赖并对损失软加权。
正文
Abstract:GRPO-style reinforcement learning has advanced vision-language reasoning, but its sequence-level rewards provide limited credit assignment for individual reasoning tokens. On-policy distillation (OPD) addresses this issue by providing dense supervision on student-generated trajectories, yet standard OPD assigns similar importance to all teacher-preferred tokens, including template phrases and visually irrelevant continuations. We introduce Visual Counterfactual On-Policy Distillation (VC-OPD), a grounding-aware distillation method that prioritizes tokens whose correctness depends on image-specific evidence. After a teacher-supervised warm start that stabilizes response formats and reduces noisy rollouts, VC-OPD performs on-policy distillation with two teacher evaluations for each student trajectory: one under the original image and one under a counterfactual image with corrupted instance-specific evidence. The original-image teacher distribution determines the distillation target, while the original--counterfactual teacher gap estimates token-level visual dependence and softly reweights the loss. To avoid unstable response-level update scales, VC-OPD further applies mass-preserving normalization to the token weights. This design preserves the dense supervision of OPD while shifting learning capacity toward visually grounded reasoning tokens. Experiments on multimodal reasoning benchmarks show that VC-OPD improves visual reasoning performance and produces more effective token-level distillation.
| Subjects: | Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7; I.2.6; I.4.8 |
| Cite as: | arXiv:2610.14939 [cs.AI] |
| (or arXiv:2610.14939v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14939 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiahe Feng [view email]
[v1]
Sat, 10 Oct 2026 14:10:43 UTC (4,768 KB)
来源：arXiv：cs.AI · arxiv.org