arXiv：cs.AI· Yibo Wang, Yuanxin Liu, Zhaoran Wang·· 20 小时前
VSR：面向可验证奖励强化学习的方差最优状态重采样
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
AI 导读
论文将可验证奖励强化学习的 rollout 预算浪费问题转化为采样分布问题，给出闭式 Neyman 分配 μ_var^π(s) ∝ d^π(s)σ_π(s)，并据此提出 VSR（Variance-driven State Resampling），这是对 GRPO 的插件式改动，使用前缀回放缓冲区和零成本方差代理。
正文
Abstract:Reinforcement learning with verifiable rewards spends much of its rollout budget on states that yield no learning signal. We recast this as a question about the sampling distribution itself: under a fixed rollout budget, which distribution over states minimizes the variance of the policy-gradient estimator? The answer admits a closed-form Neyman allocation, $\mu_{\mathrm{var}}^\pi(s) \propto d^\pi(s)\,\sigma_\pi(s)$, which connects naturally to GRPO's group-relative advantage and induces an emergent reverse curriculum as the policy improves, with no externally imposed schedule. Building on this principle, we propose VSR (Variance-driven State Resampling), a plug-in modification to GRPO using a prefix replay buffer and a zero-cost variance proxy. On mathematical reasoning benchmarks and long-horizon agentic tasks (Sokoban, BabyAI), VSR outperforms GRPO and GiGPO on most benchmarks with fewer generated tokens per update.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15888 [cs.LG] |
| (or arXiv:2610.15888v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15888 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yibo Wang [view email]
[v1]
Sun, 11 Oct 2026 20:09:24 UTC (6,888 KB)
来源：arXiv：cs.AI · arxiv.org