arXiv：cs.CL· Rahul Murali Shankar, Titus von der Malsburg, Sebastian Pad\'o·· 16 小时前
现成语言-视觉编码器可预测视觉世界实验中的注视偏好
Off-the-Shelf Language-Vision Encoders Predict Human Gaze Preferences in Visual World Experiments
AI 导读
一项研究提出对简单双模态 CLIP bi-encoder 应用归因方法来推导目标偏好，从而预测视觉世界研究中的注视模式。该方法无需生成式架构、无需微调，尽管未针对该任务训练，仍能定性复现一项经典英语视觉世界研究中的人类预测性加工注视模式。
正文
Abstract:A central question for computational psycholinguistics is to what extent neural language models can serve as cognitively adequate models of human language processing. However, work has been overwhelmingly focused on the unimodal case of written or spoken language. In contrast, multimodal paradigms, like visual world experiments that present participants with both visual and linguistic input simultaneously, have been neglected. In this paper, we present a novel approach that predicts patterns of gaze preferences in visual world studies. To do so, it derives target preferences by applying an attribution method to a simple bi-modal CLIP bi-encoder. We demonstrate the ability of this approach to qualitatively replicate the gaze patterns found in a seminal English visual world study demonstrating human predictive processing. It does so without a generative architecture and without the need for fine-tuning, despite not being trained for the task.
| Subjects: | Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2608.07282 [cs.CL] |
| (or arXiv:2608.07282v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.07282 arXiv-issued DOI via DataCite |
Submission history
From: Sebastian Pado [view email]
[v1]
Fri, 7 Aug 2026 14:42:51 UTC (752 KB)
[v2]
Wed, 14 Oct 2026 06:32:56 UTC (869 KB)
来源：arXiv：cs.CL · arxiv.org