Agentic Real2Sim：用视觉语言智能体实现基于物理的世界建模
Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents
Agentic Real2Sim 是一个用视觉语言智能体将真实物体-机器人交互录像转化为可仿真"数字孪生"的框架，并打通下游策略微调与评估。它覆盖刚体操作、可形变物体交互与人形运动三类场景，智能体决策可用开放权重 VLM 后端驱动，成本仅为前沿模型的一小部分，转化成功率却相当。项目含代码已公开。
Authors:Guanxiong Chen, Qianjun Xia, Jiawei Peng, Heng Zhang, Pengyu Jing, Bole Ma, Justin Qian, Yixian Cheng, Ziyi Jiao, Bingyang Zhou, Yiduo Qu, Luoxin Ye, Kaifeng Zhang, Kunyi Wang, Weijia Zeng, Yunuo Chen, Pengzhi Yang, Ziqiu Zeng, Siyuan Luo, Huamin Wang, Chao Liu, Alan Yuille, Fan Shi, Changxi Zheng, Yunzhu Li, Chenfanfu Jiang, Peter Yichen Chen
Abstract:Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on brittle workflow glue across visual perception tools and simulators: manual tuning of visual foundation models, mesh cleanup, coordinate frame alignments, etc. We introduce \textit{Agentic Real2Sim}, a framework for generalized physical world modeling with vision-language agents that converts a real-world recording of object-robot interaction into a simulatable episodic twin, and connects the resulting twin to downstream policy fine-tuning and evaluation. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining a comparable conversion success rate. The framework further supports custom scene conversion, fine-tuning of a pretrained policy with data generated from converted episodes, and works effectively as a surrogate for real-world policy evaluation. The project site, including code is available at this https URL.
| Comments: | Ack and RW updates |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.19190 [cs.RO] |
| (or arXiv:2607.19190v5 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2607.19190 arXiv-issued DOI via DataCite |
Submission history
From: Guanxiong Chen [view email]
[v1]
Tue, 21 Jul 2026 15:23:38 UTC (5,364 KB)
[v2]
Wed, 22 Jul 2026 03:43:37 UTC (5,364 KB)
[v3]
Fri, 24 Jul 2026 08:16:46 UTC (5,364 KB)
[v4]
Wed, 16 Sep 2026 05:10:19 UTC (4,387 KB)
[v5]
Sun, 11 Oct 2026 10:33:57 UTC (4,390 KB)
来源：arXiv：cs.AI · arxiv.org