arXiv：cs.AI· Yingyu Shan, Yuhang Guo, Zihao Cheng, Zeming Liu, Xiangrong Zhu, Xinyi Wang, Jiashu Yao, Wei Lin, Hongru Wang, Heyan Huang·· 3 小时前
SC-GRPO：以自条件信用分配提升可验证奖励强化学习
Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
AI 导读
针对 GRPO 对所有 token 均匀分配信用、浪费梯度的问题，研究者提出 SC-GRPO（Self-Conditioned GRPO），利用模型在自身已验证轨迹条件下的逐 token KL 散度作为 GRPO 梯度的乘性权重。
正文
Abstract:Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform credit across all tokens, wasting gradient on routine tokens while under-crediting pivotal reasoning steps. Existing token-level credit assignment methods require resources beyond the model's own rollouts. GRPO variants rely on process reward models or ground-truth answers. Knowledge distillation assigns credit through per-token divergence but requires external teachers (On-Policy Distillation) or privileged information (On-Policy Self Distillation). However, these dependencies limit applicability in the pure RLVR setting. We observe that conditioning the model on its own verified trajectories induces a measurable per-token KL divergence between the original and conditioned distributions, and prove that distilling from a self-teacher constructed by verified trajectories leads to infeasible weighted-average solutions when multiple verified trajectories exist. We propose SC-GRPO (Self-Conditioned GRPO), which uses KL divergence mentioned before as a multiplicative weight on GRPO gradients. Across five benchmarks spanning math, code, and agentic tasks, SC-GRPO consistently outperforms 8.1% over GRPO and 5.9% over DAPO with stronger OOD performance. Moreover, SC-GRPO achieves higher performance than OPD.
| Comments: | Accepted by EMNLP 2026 Findings |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.18810 [cs.LG] |
| (or arXiv:2606.18810v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.18810 arXiv-issued DOI via DataCite |
Submission history
From: Yingyu Shan [view email]
[v1]
Wed, 17 Jun 2026 08:26:02 UTC (4,312 KB)
[v2]
Thu, 8 Oct 2026 09:46:38 UTC (4,313 KB)
来源：arXiv：cs.AI · arxiv.org