arXiv：cs.LG· Kunwang Zheng, Xinyu Sun, Yu Cao, Shuocheng Li, Lei Cheng, Ying Liu, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen·· 4 小时前
RELAY：面向评分标准强化学习的自适应交接选择性自蒸馏方法
RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
AI 导读
RELAY 是一种面向评分标准（rubric）强化学习的选择性自蒸馏方法，通过优势与奖励筛选 rollout，并用蒸馏损失监测教师模型退出的时机。阶段一将冻结教师的选择性蒸馏与 GRPO 结合，平台期检测器随后移除教师，阶段二仅用 GRPO 继续训练；检测器在一个模型-领域组合上校准后直接复用于另外三个，触发步数在 122 至 171 之间。
正文
Abstract:Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.
| Comments: | 17 pages, 8 figures, 13 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.17623 [cs.LG] |
| (or arXiv:2610.17623v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17623 arXiv-issued DOI via DataCite |
Submission history
From: Kunwang Zheng [view email]
[v1]
Tue, 13 Oct 2026 02:45:54 UTC (1,718 KB)
[v2]
Wed, 14 Oct 2026 05:22:37 UTC (1,718 KB)
来源：arXiv：cs.LG · arxiv.org