从视觉事件生成自然语言：研究现状与关键开放问题

View PDF HTML (experimental)

Abstract:In recent years, a substantial body of work in visually grounded natural language processing has focused on real-life multimodal scenarios such as describing content depicted in images or videos. However, comparatively less attention has been devoted to studying the nature and degree of interaction between the different modalities in these scenarios. In this paper, we argue that any task dealing with natural language generation from sequences of images or frames is an instance of the broader, more general problem of modeling the intricate relationships between visual events unfolding over time and the features of the language used to interpret, describe, or narrate them. Therefore, solving these tasks requires models to be capable of identifying and managing such intricacies. We consider five seemingly different tasks, which we argue are compelling instances of this broader multimodal problem. Subsequently, we survey representative modeling and evaluation approaches adopted for these tasks in recent years and examine the common set of challenges these tasks pose. Building on this perspective, we identify key open questions and propose several research directions for future investigation.

Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG)
Cite as:	arXiv:2502.13034 [cs.CL]
	(or arXiv:2502.13034v4 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2502.13034 arXiv-issued DOI via DataCite

Submission history

From: Aditya K Surikuchi [view email]
[v1] Tue, 18 Feb 2025 16:48:18 UTC (2,470 KB)
[v2] Sun, 25 May 2025 13:02:56 UTC (11,304 KB)
[v3] Tue, 19 Aug 2025 20:12:30 UTC (11,277 KB)
[v4] Sun, 11 Oct 2026 18:19:04 UTC (11,605 KB)