S-trace：面向 RLVR 的选择性后继资格迹方法

View PDF HTML (experimental)

Abstract:Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of large language models. However, widely used critic-free algorithms such as Group Relative Policy Optimization (GRPO) suffer from coarse-grained temporal credit assignment, as they assign the same trajectory-level advantage to all tokens and fail to distinguish the contributions of different reasoning steps. To address this challenge, we propose Selective Successor Eligibility Traces (S-trace), an critic-free eligibility trace method that enables more effective credit assignment. S-trace introduces successor importance reweighting to recover downstream credit signals under PPO-style clipping and propagates them to selected historical tokens through entropy-based token selection. Theoretically, we establish an upper bound on its discrepancy from the raw eligibility traces-based policy gradient estimator, jointly controlled by the KL divergence and the trace decay factor $\lambda$. Experiments show that S-trace achieves absolute gains in average pass@16 of 1.80\% and 6.17\% over GRPO, and 2.32\% and 0.87\% over trace-style GRPO($\lambda$), on Qwen3-1.7B and Qwen3-4B, respectively, while maintaining robust gains on Qwen3-8B and producing shorter responses.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2605.05965 [cs.LG]
	(or arXiv:2605.05965v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.05965 arXiv-issued DOI via DataCite

Submission history

From: Chaoli Mou [view email]
[v1] Thu, 7 May 2026 10:11:51 UTC (626 KB)
[v2] Tue, 13 Oct 2026 13:21:17 UTC (287 KB)