arXiv：cs.AI· Zonghuan Li, Hayley Hung, Chirag Raman·· 16 小时前
GesBench：多模态语言模型何时、何处感知社交含义？
GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?
AI 导读
研究者推出 GesBench，一个基于人类标注的多模态 LLM 线索级社交感知基准，用累积子话语片段测量模型何时形成对说话人意图的稳定判断，并用人类注视引导的破坏实验检验人类关注区域对模型判断的功能重要性。该基准由 6000 多个话语级片段构建，覆盖脚本与非脚本对话数据集。在开源与闭源多模态模型上均发现明显的人机差距：人类更早形成预测性理解且修正更频繁，模型则常在不同片段间含糊或重复。
正文
Abstract:Human social perception is incremental: we infer intent before an utterance is complete, revise our interpretation as new cues arrive, and attend to specific facial, gestural, or prosodic signals that shape social meaning. Current multimodal LLM benchmarks largely evaluate final-answer understanding over complete episodes, rewarding post-hoc recognition of social meaning rather than the unfolding perception required for real-time interaction. Drawing on interactionally embedded Gestalt principles of multimodal human communication, we introduce GesBench: a human-grounded benchmark for cue-level social perception in multimodal LLMs. GesBench uses cumulative sub-utterance clips to measure when models form stable interpretations of speaker intention and social affordance, and human gaze-guided corruption to evaluate whether regions attended by humans are functionally important for model judgments. Built from more than 6000 utterance-level clips from scripted and unscripted conversational datasets, GesBench includes human annotations of unfolding judgments and gaze on a representative subset. Across open and closed multimodal models, we find a clear human-model gap: humans form predictive understandings early and revise more often, while models often hedge or repeat across different clips. GesBench shifts social evaluation from whether models eventually recognize meaning to when and where that meaning is formed. Our codes are publicly available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15129 [cs.AI] |
| (or arXiv:2610.15129v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15129 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zonghuan Li [view email]
[v1]
Sat, 10 Oct 2026 19:15:11 UTC (7,086 KB)
来源：arXiv：cs.AI · arxiv.org