arXiv：cs.AI· Lihao Zheng, Zhenwei Shao, Zhou Yu, Yan Yang, Haoyi Sun, Qian Wang, Ning Mao, Pan Zhou·· 22 小时前
VAttnPO：视觉注意力引导的策略优化
VAttnPO: Visual Attention-Guided Policy Optimization
AI 导读
VAttnPO 用答案条件下的视觉注意力修正 token 级信用分配，同时保留 GRPO 的轨迹平均优势。该方法在 Qwen3-VL-8B-Instruct 上将六任务图像 AVE 从 GRPO 的 70.97 提升至 73.50，并在结构化、视频和 MME 指标上全面领先；在 Qwen2.5-VL-7B-Instruct 和 InternVL3-8B-Instruct 上的增益同样成立。
正文
Abstract:Multimodal reinforcement learning with verifiable rewards commonly assigns one sequence advantage to an entire response. This outcome-level signal distinguishes overall response quality but leaves changes in visual-evidence use across generated tokens implicit. We introduce VAttnPO (Visual Attention-Guided Policy Optimization), which uses answer-conditioned visual attention to refine token-level credit while preserving the verifier-derived trajectory-mean advantage. A frozen old policy constructs a visual reference by teacher forcing the known correct answer and produces a visual distribution for each rollout token. VAttnPO measures their agreement in visual-attention space using Jensen--Shannon divergence, rather than reading guidance from teacher output-token distributions or introducing an attention-matching objective. Trajectory centering, relative-position weighting, and credit balancing turn this agreement into a zero-mean, position-aware correction to the GRPO sequence advantage. Token intervention, visual-index permutation, and trajectory analysis provide diagnostic evidence linking this signal to model behavior, visual correspondence, and generation position. On Qwen3-VL-8B-Instruct, VAttnPO raises six-task image AVE from 70.97 with GRPO to 73.50 and scores higher on every reported structured, video, and MME metric. Gains over GRPO also hold on Qwen2.5-VL-7B-Instruct and InternVL3-8B-Instruct.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16735 [cs.CV] |
| (or arXiv:2610.16735v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16735 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lihao Zheng [view email]
[v1]
Mon, 12 Oct 2026 13:04:45 UTC (1,847 KB)
来源：arXiv：cs.AI · arxiv.org