Candidate Set Visual Grounding（CSVG）：对话歧义识别新任务与 CandVG 基准

View PDF HTML (experimental)

Abstract:Interpreting an ambiguous utterance in real-world interactions often requires context beyond a single utterance. When the utterance refers to objects in a visual scene, it is necessary to maintain and update the set of plausible referents as context accumulates, without assuming that the available information identifies a unique object. However, prior work on interactive visual grounding, which localizes objects referred to by linguistic expressions, has focused on identifying a unique object once sufficient dialogue context becomes available. To address this gap, we propose Candidate Set Visual Grounding (CSVG), a task for estimating the set of objects consistent with the given dialogue context. We further introduce CandVG, a benchmark that evaluates how the candidate set changes across dialogue turns. Our evaluation of vision-language models shows that stronger visual grounding capability does not necessarily yield better CSVG performance. In particular, models have difficulty predicting the complete candidate set, owing to challenges in candidate generation and premature commitment to a small set of candidates in early turns. Nevertheless, we find that this limitation can be partially mitigated in the evaluated large-scale models using instructions designed for CSVG.

Comments:	Data: this https URL, Code: this https URL
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
Cite as:	arXiv:2610.19723 [cs.CL]
	(or arXiv:2610.19723v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.19723 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Shun Inadumi [view email]
[v1] Wed, 14 Oct 2026 07:48:25 UTC (2,568 KB)