HuggingFace Daily Papers·· 1 天前
RynnWorld-Latent：以通用潜在动作为条件的跨具身世界模型
RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
AI 导读
RynnWorld-Latent 是一个跨具身视频世界模型，用 RynnLAM 提出的 3D 接地潜在动作作为共享条件接口，无需统一原生动作空间即可联合预训练人类交互视频、机器人演示与仿真数据。
正文 · 原文
Authors:Haoyu Zhao, Yinzhou Tang, Zixuan Wang, Minghao Zhu, Dongchi Huang, Xingyue Zhao, Qin Zhao, Kehan Li, Siteng Huang, Xin Li, Zhenyu Zeng, Mingxiu Chen, Deli Zhao, Zhongyu Li
Abstract:Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.
| Comments: | Homepage this https URL, Github this https URL |
| Subjects: | Robotics (cs.RO) |
| Cite as: | arXiv:2610.36791 [cs.RO] |
| (or arXiv:2610.36791v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.36791 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haoyu Zhao [view email]
[v1]
Tue, 27 Oct 2026 07:54:16 UTC (19,906 KB)
来源：HuggingFace Daily Papers · arxiv.org