arXiv：cs.CL· Derek Lilienthal, Manisha Mukherjee, Sameera Horawalavithana·· 6 小时前
视觉语言模型物理推理的奖励设计：外部奖励与内部注意力奖励对比
Reward Design for Physical Reasoning in Vision-Language Models
AI 导读
研究对比了视觉语言模型物理推理中的外部奖励与内部注意力奖励：外部奖励由规则化格式与答案正确性检查或 LLM 评判提供，带来大部分准确率提升，奖励推理 token 时推理链更连贯。内部注意力奖励无需标注或评判，能改善空间关系推理，且在 Molmo 2 上以零额外标注成本与格式、准确率奖励叠加后提升平均准确率，并泛化到分布外任务。
正文
Abstract:Physical reasoning over images requires integrating perception, domain knowledge, and multi-step symbolic inference, yet Vision-Language Models (VLMs) fall far short of human performance on physics benchmarks. Reinforcement learning can improve reasoning via reward signals, but it remains unclear whether these signals must come from outside the model or can be derived from the model itself. We compare external rewards, which rely on an outside verifier (rule-based format and answer-correctness checks, or an LLM-judge rubric), with an internal attention-based reward computed from the model's own attention over image regions, requiring no labels or judge. We find that external rewards drive most of the accuracy gains, while preserving more coherent reasoning when rewarded for reasoning tokens. The internal attention-based reward, provides an annotation-free grounding signal that improves spatial-relation reasoning and is complementary to external rewards: adding it to format and accuracy rewards, at no additional annotation cost, improves average accuracy on Molmo~2 while generalizing to out-of-distribution tasks. Our results suggest that internal attention rewards can supplement external supervision and lead to improved physical reasoning.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2604.13993 [cs.AI] |
| (or arXiv:2604.13993v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.13993 arXiv-issued DOI via DataCite |
Submission history
From: Derek Lilienthal [view email]
[v1]
Wed, 15 Apr 2026 15:36:26 UTC (1,797 KB)
[v2]
Wed, 14 Oct 2026 15:56:56 UTC (2,736 KB)
来源：arXiv：cs.CL · arxiv.org