arXiv：cs.AI· Joseph Amigo, Rooholla Khorrambakht, Nicolas Mansard, Ludovic Righetti·· 22 小时前
耦合局部与全局世界模型：高效一阶强化学习新方法
Coupled Local and Global World Models for Efficient First Order RL
AI 导读
研究者提出一种绕过模拟器的方法，直接在从真实机器人交互中学习的世界模型内训练 RL 策略，核心是解耦一阶梯度（FoG）方法：全局世界模型生成精确前向轨迹，轻量隐空间代理模型近似局部动力学以高效计算梯度。在 Push-T 操作任务上，该方法样本效率显著优于 PPO，并在四足机器人的自我中心物体操作任务上得到验证，论文已被 CoRL 2026 接收。
正文
Abstract:World models offer a promising avenue for more faithfully capturing complex dynamics, including contacts and non-rigidity, as well as complex sensory information, such as visual perception, in situations where standard simulators struggle. However, these models are computationally complex to evaluate, posing a challenge for popular RL approaches that have been successfully used with simulators to solve complex locomotion tasks but yet struggle with manipulation. This paper introduces a method that bypasses simulators entirely, training RL policies inside world models learned from robots' interactions with real environments. At its core, our approach enables policy training with large-scale diffusion models via a novel decoupled first-order gradient (FoG) method: a full-scale world model generates accurate forward trajectories, while a lightweight latent-space surrogate approximates its local dynamics for efficient gradient computation. This coupling of a local and global world model ensures high-fidelity unrolling alongside computationally tractable differentiation. We demonstrate the efficacy of our method on the Push-T manipulation task, where it significantly outperforms PPO in sample efficiency. We further evaluate our approach through an ego-centric object manipulation task with a quadruped. Together, these results demonstrate that learning inside data-driven world models is a promising pathway for solving hard-to-model RL tasks in image space without reliance on hand-crafted physics simulators.
| Comments: | Accepted at the 10th Conference on Robot Learning (CoRL 2026). Project website: this https URL |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.06219 [cs.RO] |
| (or arXiv:2602.06219v3 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2602.06219 arXiv-issued DOI via DataCite |
Submission history
From: Joseph Amigo [view email]
[v1]
Thu, 5 Feb 2026 21:57:41 UTC (8,562 KB)
[v2]
Mon, 1 Jun 2026 20:55:42 UTC (37,961 KB)
[v3]
Sun, 11 Oct 2026 21:41:24 UTC (26,535 KB)
来源：arXiv：cs.AI · arxiv.org