arXiv：cs.AI· Anum Afzal, Yuki Saito, Hiroya Takamura, Katsuhito Sudoh, Shinnosuke Takamichi, Graham Neubig, Florian Matthes, Tatsuya Ishigaki·· 7 小时前AI 评分42
多模态 LLM 实时游戏视频解说：基于暂停感知的解码方法
Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches
AI 导读
研究提出两种基于提示的解码策略实现多模态 LLM 实时游戏视频解说：固定间隔法与动态间隔解码法，后者根据上一句解说的估计时长调整下一次预测时机，无需微调即可实现暂停感知生成。在赛车与格斗游戏的日语和英语数据集上，动态间隔解码生成的解说在时间与内容上更贴近人类表述。团队开源了多语言基准数据集、训练模型及实现代码。
正文
Abstract:Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in domains such as sports, esports, and livestreaming. Commentary generation involves two essential decisions: what to say and when to say it. While recent prompting-based approaches using multimodal large language models (MLLMs) have shown strong performance in content generation, they largely ignore the timing aspect. We investigate whether in-context prompting alone can support real-time commentary generation that is both semantically relevant and well-timed. We propose two prompting-based decoding strategies: 1) a fixed-interval approach, and 2) a novel dynamic interval-based decoding approach that adjusts the next prediction timing based on the estimated duration of the previous utterance. Both methods enable pause-aware generation without any fine-tuning. Experiments on Japanese and English datasets of racing and fighting games show that the dynamic interval-based decoding can generate commentary more closely aligned with human utterance timing and content using prompting alone. We release a multilingual benchmark dataset, trained models, and implementations to support future research on real-time video commentary generation.
| Comments: | Accepted at LREC2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.02655 [cs.CL] |
| (or arXiv:2603.02655v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.02655 arXiv-issued DOI via DataCite |
Submission history
From: Tatsuya Ishigaki [view email]
[v1]
Tue, 3 Mar 2026 06:39:04 UTC (6,051 KB)
[v2]
Tue, 6 Oct 2026 13:18:40 UTC (11,506 KB)
来源：arXiv：cs.AI · arxiv.org