arXiv：cs.LG· Octi Zhang, Mateo Guaman Castro, Patrick Yin, Ignacio Dagnino, Abhishek Gupta, Rosario Scalise, Byron Boots·· 3 小时前
解决超大规模机器人强化学习的探索瓶颈：Success Guided Sampling
A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control
AI 导读
研究者提出 Success Guided Sampling（SGS），一种将 RL 训练集中在策略能力前沿任务配置上的自适应采样器，缓解了均匀采样导致的学习信号浪费。在最多 2^20（超百万）个并行环境的实验中，SGS 让 RL 解决了此前方法无法完成的多地形四足locomotion和接触密集装配任务，并将学到的操作策略蒸馏为基于 RGB 的策略，在真实硬件上实现零样本迁移。
正文
Abstract:General-purpose robots must perform a wide range of tasks from agile locomotion to dexterous manipulation. While sim-to-real reinforcement learning (RL) has proven to be a useful tool for this goal, current RL pipelines depend on engineering-heavy, per-task structural priors such as shaped rewards and demonstrations. Recent work has shown that diverse simulator resets, combined with massively parallel simulation, can alleviate much of this engineering burden on several manipulation problems. However, we find that naively scaling this paradigm to more precise or dynamic problems remains non-trivial. While simulator resets can help with exploration, uniformly sampling over this distribution wastes a growing fraction of learning experience on task configurations the policy has already mastered or cannot yet attempt. This makes it challenging to see the expected benefits of scaling parallel environments for RL, since much of the learning signal in a batch is wasted during learning. To mitigate this, we introduce Success Guided Sampling (SGS), a simple adaptive sampler that concentrates RL training on task configurations around the frontier of the policy's capabilities. Doing so allows large-scale simulated RL to make the most out of the experience in a batch, enabling much more effective scaling to large-scale parallel simulation. Across experiments using up to $2^{20}$ (over one million) parallel environments, SGS enables RL to solve challenging multi-terrain quadruped locomotion and contact-rich assembly tasks that prior methods fail to solve. Finally, we distill the learned manipulation policies into RGB-based policies and demonstrate zero-shot transfer to several challenging assembly tasks on real hardware. Project website: this https URL.
| Comments: | CoRL 2026. Project website: this https URL |
| Subjects: | Robotics (cs.RO); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12465 [cs.RO] |
| (or arXiv:2610.12465v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12465 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mateo Guaman Castro [view email]
[v1]
Thu, 8 Oct 2026 17:59:50 UTC (6,685 KB)
来源：arXiv：cs.LG · arxiv.org