Plan, Watch, Recover：主动式流程辅助的基准与架构
Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance
研究提出 EgoProactive 智能眼镜数据集与 Pro²Bench 跨域基准，评测主动式多模态助手的干预时机、流程完成度与响应质量。零样本下，无任何前沿或开源权重模型能在首次出错前完成超过 26% 的流程干预。所提 PWR（Plan, Watch, Recover）在上下文中维护显式计划状态并在每次干预后修订，三个开源骨干上平均流程完成率从 20% 提升至 27%。
Authors:Kaustav Kundu, Ritvik Shrivastava, Maxim Arap, Nanshu Wang, Xianhui Zhu, Quintin Fettes, Gautam Tiwari, Parth Suresh, Prateek Singhi, Théo Moutakanni, Alejandro Castillejo Munoz, Allen Bolourchi, Pascale Fung, Pinar Donmez, Babak Damavandi, Anuj Kumar, Seungwhan Moon
Abstract:Proactive multimodal assistants must decide when guidance is needed and what to say, while tracking progress through extended procedures and helping users recover when they depart from the expected plan. Progress is limited by benchmarks that score isolated clips, cover narrow domains, or omit realistic deviations. We introduce EgoProactive, a smart-glasses dataset of procedures across four domains in which users commit step-level deviations and recover from them, paired with reference guidance. We combine it with five re-annotated egocentric and instructional datasets to form Pro$^{2}$Bench, a cross-domain benchmark that scores intervention timing, procedure completion, and response quality. Zero-shot, no frontier or open-weight model completes more than 26% of a procedure's interventions before its first error. We propose PWR (Plan, Watch, Recover), which keeps an explicit plan state in the model's context and revises it after every intervention, including recovery steps after a deviation. Across three open-weight backbones, training on Pro$^{2}$Bench without a plan barely changes performance, while adding the dynamic plan raises average procedure completion from 20% to 27% and improves turn-level decisions and response quality for every backbone. The gains hold on held-out datasets and on deviations never seen in training.
| Comments: | 39 pages, 13 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.04970 [cs.CV] |
| (or arXiv:2606.04970v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.04970 arXiv-issued DOI via DataCite |
Submission history
From: Ritvik Shrivastava [view email]
[v1]
Wed, 3 Jun 2026 14:52:03 UTC (9,458 KB)
[v2]
Mon, 12 Oct 2026 23:29:05 UTC (6,604 KB)
来源：arXiv：cs.AI · arxiv.org