HuggingFace Daily Papers·· 1 天前
长时程 RL 需要 Muon：单 rollout 异步训练配方提升仓库级问题解决成功率
Long-Horizon RL Needs Muon
AI 导读
一项工作提出面向长时程智能体的单 rollout 异步训练配方，将 Muon 优化器、token 平均损失、截断重要性采样以及长度与工具错误辅助惩罚项组合使用，无需等待同一问题的多条轨迹即可异步学习。作者在两个模型族上评估仓库级问题解决任务，结果显示完整配方的训练成功率高于基线。该工作共 7 页、5 图，仍为进行中的研究。
正文
Abstract:Long-horizon agents must coordinate reasoning, tool use, and environment feedback over extended interactions. Learning from sparse outcome rewards requires weighting trajectories of different lengths and updating the policy while rollouts are still being collected. We examine a single-rollout asynchronous recipe combining Muon, token-averaged loss, truncated importance sampling, and auxiliary length and tool-error penalties. Single-rollout sampling allows asynchronous learning without waiting for multiple trajectories for the same issue. We study optimizer choices and recipe components for repository-level issue resolution across two model families. The evaluated training runs show higher training success with the full recipe than with the baselines.
| Comments: | 7 pages, 5 figures. Work in progress |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20256 [cs.LG] |
| (or arXiv:2610.20256v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20256 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kai Ruan [view email]
[v1]
Wed, 14 Oct 2026 14:25:54 UTC (283 KB)
来源：HuggingFace Daily Papers · arxiv.org