HuggingFace Daily Papers·· 17 小时前
Reviso：让视频描述模型学会有重点地观察视觉证据
Reviso: Not All Visual Evidence Deserves Equal Attention in Video Captioning
AI 导读
研究提出智能体视频描述框架 Reviso，从低帧率全局预览构建 Context–Focus–Points 分层观察计划，选择性回看关键片段并调用工具细查，再整合全局与局部信息生成描述。
正文 · 原文
Authors:Tengfei Liu, Yang Shi, Yuran Wang, Yuqi Tang, Yizhuo Jia, Zhuoran Zhang, Qixun Wang, Shujie Li, Xuanyu Zhu, Xinyu Liu, Minghui Guo, Shixuan Liu, Haotian Wang, Yuanxing Zhang, Pengfei Wan
Abstract:Multimodal large language models (MLLMs) can generate detailed video captions, yet they may overemphasize minor details while overlooking essential context and key events. We argue that not all visual evidence deserves equal attention, as the importance of visual information depends on the content of each video. Motivated by this observation, we propose Reviso, an agentic video captioning framework that learns what to observe and how to gather relevant visual evidence. From a low-frame-rate global preview, Reviso constructs a hierarchical Context--Focus--Points (CFP) observation plan that identifies the video context, prioritizes key aspects, and defines concrete observation targets and corresponding tools. Guided by this plan, Reviso performs goal-directed active perception by selectively revisiting temporal segments and combining tools for fine-grained inspection. It interprets the acquired evidence, updates its plan as needed, and integrates global and local information to generate the final caption. To learn this observation strategy, we develop a verification-guided pipeline that constructs high-quality training trajectories through factual verification and teacher-guided refinement. We further train Reviso using Group Relative Policy Optimization (GRPO), incorporating a key-fact reward for caption quality, a Plan Alignment Reward (PAR) for identifying important observation targets, and an Unproductive Tool-Call Penalty (UTCP) that adjusts observation-cost penalties based on caption quality. Experiments across four video captioning benchmarks show that Reviso outperforms the evaluated open-source baselines, while its generated captions better support downstream video question answering. These findings highlight the effectiveness and potential of priority-guided active observation for video captioning.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2611.03602 [cs.CV] |
| (or arXiv:2611.03602v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2611.03602 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tengfei Liu [view email]
[v1]
Mon, 2 Nov 2026 18:41:14 UTC (16,191 KB)
来源：HuggingFace Daily Papers · arxiv.org