arXiv：cs.AI· Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou, Lingjun Mao, Ruobing Han, Jinglin Cao, Wenpeng Xu, Lukun He, Minghao Fu, Fan Feng, Biwei Huang·· 13 小时前
Video2World：用编码智能体从具身视频自动构建交互世界模型
Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
AI 导读
研究者提出 Video2World 基准，将"自主视频转仿真"定义为软件工程任务：智能体观察具身视频，构建对应仿真环境与机器人行为，并通过执行反馈迭代改进。该基准包含来自 189 段机器人与人类演示视频的 222 个重建实例，从几何保真度、动态保真度和功能正确性三个维度评估。
正文
Authors:Jinzhou Tang, Zijun Zhang, Jing Yang, Yuchen Yan, Kun Zhou, Lingjun Mao, Ruobing Han, Jinglin Cao, Wenpeng Xu, Lukun He, Minghao Fu, Fan Feng, Biwei Huang
Abstract:Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manual environment construction and calibration. We study whether frontier foundation models and coding agents can automate this process end to end. We formulate \emph{autonomous video-to-simulation} as a software engineering task in which an agent observes an embodied video, constructs the corresponding simulated environment and robot behavior, and iteratively refines the result through execution feedback. To evaluate this capability, we introduce \textbf{Video2World}, a benchmark comprising 222 reconstruction instances derived from 189 robot and human demonstration videos. Video2World measures reconstructed worlds along geometric fidelity, dynamic fidelity, and functional correctness, capturing spatial perception, physical reasoning, and executable interaction. Evaluating 9 frontier coding-agent systems reveals a sharp improvement in Task success beginning with Claude Opus 5, rising from below 5\% to over 15\%, while substantial gaps to human-assisted reconstruction remain. We further find that worlds that look better could work worse: better visual fidelity does not always lead to higher task success. This echoes the broader gap between perceptual realism and factual correctness observed in generative models.
| Comments: | Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2610.04432 [cs.CV] |
| (or arXiv:2610.04432v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.04432 arXiv-issued DOI via DataCite |
Submission history
From: Jinzhou Tang [view email]
[v1]
Sat, 3 Oct 2026 10:46:52 UTC (42,988 KB)
[v2]
Wed, 7 Oct 2026 16:40:14 UTC (31,270 KB)
[v3]
Fri, 9 Oct 2026 05:21:02 UTC (31,270 KB)
来源：arXiv：cs.AI · arxiv.org