HuggingFace Daily Papers·· 2 天前
MRTPO：让每个奖励只作用于相关 token 的多奖励强化学习信用分配方法
Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL
AI 导读
针对多奖励 RL 中 GDPO 将每个奖励的 advantage 均匀铺满整个回复的问题，研究者提出 MRTPO，利用各奖励 verifier 实际检查的位置，把每个奖励的 advantage 只分配给与之相关的 token。
正文
Abstract:Language models are increasingly trained against several verifiable rewards at once, such as the correctness and the format of a tool call, or the individual constraints of a complex instruction. Reward-decoupled normalization (GDPO) normalizes each reward separately so that different reward combinations keep distinct advantages, but it still spreads every reward's advantage evenly over the whole response, although each verifier checks only a few positions. We propose MRTPO, which assigns the advantage of each reward to the tokens relevant to it, using the information its verifier checks, without auxiliary models or token-level annotations. For each reward, the policy replays its own response with and without that reward's verifier information, and the per-token Jensen-Shannon distance between the two next-token distributions measures each token's relevance and reassigns the advantage. On text-to-SQL, multi-constraint instruction following and tool calling, MRTPO converges faster and achieves better performance than the compared methods.
| Comments: | 15 pages, 8 figures. Project page: this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.21439 [cs.LG] |
| (or arXiv:2610.21439v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21439 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haotian Zhai [view email]
[v1]
Thu, 15 Oct 2026 07:47:04 UTC (2,169 KB)
来源：HuggingFace Daily Papers · arxiv.org