arXiv：cs.AI· Ziqi Wen, Parsa Madinei, Miguel P. Eckstein·· 22 小时前
研究揭示人类与 VLM 场景描述之间的语义显著差距
Revealing the Gap in Human and VLM Scene Description through Counterfactual Semantic Saliency
AI 导读
研究者提出 Counterfactual Semantic Saliency（CSS）黑盒框架，通过因果消融物体衡量场景语义变化，以评估 VLM 与人类的语义对齐。测试覆盖 307 个复杂自然场景、1,306 个反事实变体及 16,603 条人类心理物理学基线回答，发现模型比人类更依赖大物体、中心物体和高显著度物体，对人物的依赖则低于人类。模型的尺寸偏差是解释人机语义分歧变化的主要因素。
正文
Abstract:Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Traditional white-box interpretability methods are inapplicable to closed-source architectures and passive metrics fail to isolate causal features. We introduce Counterfactual Semantic Saliency (CSS). This black-box, model-agnostic framework quantifies the importance of objects by measuring the semantic shift induced by their causal ablation from a scene. To evaluate AI-human semantic alignment, we tested prominent VLMs against a human psychophysics baseline comprising 16,603 valid responses across 307 complex natural scenes and 1,306 high-fidelity counterfactual variants. Our analysis reveals a pervasive scene comprehension gap: models exhibit an overreliance (relative to humans) on large objects (size bias), objects at the center of the image (center bias), and high-saliency objects. In contrast, models rely less on people in the scenes than our human participants do to describe the images. A model's size bias is a primary driver explaining variations in model-human semantic divergence. Code and data are available at this https URL .
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.13047 [cs.CV] |
| (or arXiv:2605.13047v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.13047 arXiv-issued DOI via DataCite |
Submission history
From: Ziqi Wen [view email]
[v1]
Wed, 13 May 2026 06:11:47 UTC (5,479 KB)
[v2]
Sat, 10 Oct 2026 19:45:17 UTC (6,089 KB)
来源：arXiv：cs.AI · arxiv.org