arXiv：cs.LG· Yiming Zong, Yige Wang, Xing Hu, Jiashuo Jiang, Zuo-Jun Max Shen·· 4 小时前AI 评分33
CERO：RL 后训练中如何分配 rollout 预算
CERO: Where and When to Allocate Rollouts for RL Post-Training
AI 导读
针对群体相对强化学习固定每次更新预算的做法，研究者提出 CERO——一种在线原始-对偶调度器，在整段训练周期内协调有限的 rollout 预算，自适应决定选哪些 prompt、多久复访以及每轮生成多少响应组。在相同训练响应预算下，CERO 在三个骨干模型、五个数学推理基准上均取得最高的 avg@16 宏平均。
正文
Abstract:Adaptive rollout methods for group-relative reinforcement learning typically allocate a fixed per-update budget across prompts. We instead study how to coordinate a finite rollout budget over the entire training horizon. We formulate this problem using a concave surrogate utility of cumulative prompt exposure and introduce CERO, an online primal dual scheduler for prompt admission and budget pacing. In our experiments, each admitted prompt receives a fixed-size response group. CERO instead adapts which prompts are selected, how often they are revisited across rounds, and how many groups are generated in each round. A compact Fenchel representation linearizes the dependence on cumulative exposure, while projected online gradient descent updates prompt-specific supporting slopes and a shared budget price using reward-variation feedback and budget deviations. We establish pathwise guarantees for the surrogate allocation objective against fixed-rate and same-path time-varying benchmarks, with explicit terms for proxy discrepancy and rate variation. Under matched training-response budgets, CERO attains the highest avg@16 macro-average on each of three backbones across five mathematical reasoning benchmarks. Mechanistic analyses link CERO's prompt choices to within-group reward contrast, while multi-seed ablations show gains from adaptive pacing over both uniform and preset spending schedules.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.09679 [cs.LG] |
| (or arXiv:2610.09679v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09679 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yiming Zong [view email]
[v1]
Wed, 7 Oct 2026 08:40:53 UTC (258 KB)
来源：arXiv：cs.LG · arxiv.org