arXiv：cs.AI· Long Li, Zhijian Zhou, Tianyi Wang, Weidi Xu, Zuming Huang, Wei Chu, Zhe Wang, Shirui Pan, Chao Qu, Yuan Qi·· 5 小时前
DyJR：用动态 Jensen-Shannon 回放保持可验证奖励强化学习中的多样性
DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay
AI 导读
针对 GRPO 等 on-policy 算法丢弃历史 rollout、经验回放又易导致策略过度集中的问题，研究者提出 Dynamic Jensen-Shannon Replay（DyJR）。
正文
Abstract:While Reinforcement Learning (RL) enhances Large Language Model reasoning, on-policy algorithms like GRPO are sample-inefficient as they discard past rollouts. Existing experience replay methods address this by reusing accurate samples for direct policy updates, but this can incur additional storage costs and concentrate the policy on a narrow set of solutions. We argue that historical data should prioritize sustaining local policy plasticity rather than simply reinforcing accuracy. We define local policy plasticity as a persistent ability to remain backtrackable and switchable with respect to recent successful trajectories: throughout training, the policy maintains non-negligible support on recently reachable alternatives, so it can smoothly reallocate probability mass when needed without being locked into a single dominant path. To this end, we propose Dynamic Jensen-Shannon Replay (DyJR), a simple yet effective regularization framework using a dynamic reference distribution from recent trajectories. DyJR introduces two innovations: (1) A Time-Sensitive Dynamic Buffer that uses FIFO and adaptive sizing to retain only temporally proximal samples, synchronizing with model evolution; and (2) Jensen-Shannon Divergence Regularization, which replaces direct replay policy-gradient updates with a regularization signal that discourages premature over-commitment to a single Rank-1 trajectory. Experiments on mathematical reasoning, Text-to-SQL, and coding benchmarks demonstrate that DyJR improves over GRPO as well as baselines such as RLEP and Ex-GRPO, while maintaining training efficiency comparable to the original GRPO. Furthermore, Rank-k token probability evolution and semantic-mode analyses show reduced concentration and greater retention of alternative continuations. Matched admission-objective comparisons and repeated runs provide additional evidence for the roles of DyJR's components.
| Comments: | Accepted at NeurIPS 2026. 30 pages, 4 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.16157 [cs.LG] |
| (or arXiv:2603.16157v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.16157 arXiv-issued DOI via DataCite |
Submission history
From: Long Li [view email]
[v1]
Tue, 17 Mar 2026 06:20:56 UTC (209 KB)
[v2]
Mon, 12 Oct 2026 09:12:11 UTC (560 KB)
来源：arXiv：cs.AI · arxiv.org