arXiv：cs.CL· Shubh Garg, Chen Shani·· 13 小时前
研究揭示 VLM 空间关系表征弱于词汇先验：EICS 指标与 CASP-VL 基准
When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors
AI 导读
研究提出 EICS 指标和基于 Visual Genome 场景图的 CASP-VL 基准，发现多种 VLM 架构中空间关系的表征忠实度显著低于词汇信息，即使同一模型在组合性基准上得分远高于随机水平。分析显示关系在谓词自身 token 上可识别，但句子经 EOS token 或均值池化聚合成全局表示后即丢失。这表明 VLM 未在下游所用表征中保留输入中的关系信息，构成输出评估无法发现的系统性缺陷。
正文
Abstract:Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully encode the structured relations required for such reasoning. We introduce EICS (Evaluation-Invariant Concept Stability), a metric for testing whether representations consistently encode concepts under controlled one-concept interventions, and CASP-VL (Causally controlled Spatial Perturbations for Vision-Language models), a benchmark of causally controlled spatial perturbations derived from Visual Genome scene graphs. Across multiple VLM architectures, we find that spatial relations are represented substantially less faithfully than lexical information, even when the same models score well above chance on compositional benchmarks. Further analysis shows that the relation is identifiable at the predicate's own tokens but is lost when the sentence is aggregated into a global representation, whether through the (EOS) token or by mean pooling. These findings reveal a systematic failure invisible to output-based evaluation: VLMs do not preserve, in the representation used downstream, relational information that is present in their own input.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13910 [cs.CL] |
| (or arXiv:2610.13910v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13910 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chen Shani [view email]
[v1]
Fri, 9 Oct 2026 17:50:03 UTC (83 KB)
来源：arXiv：cs.CL · arxiv.org