arXiv：cs.AI· Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang, Alborz Geramifard·· 20 小时前
超越 GRPO 与在线策略蒸馏：语言模型后训练的稀疏到稠密奖励原则
Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Training
AI 导读
研究提出“奖励密度原则”：稀疏序列级奖励适合让强模型探索更优行为，稠密 token 级教师监督更适合将其压缩进小模型，稀缺标注数据应先用在上游最强教师上。在 Qwen3-1.7B 学生模型上，经 RL 改进的 8B 教师蒸馏后在 MATH 达 79.3%、AIME 2024 达 25.2%，均高于直接 GRPO 的 75.9% 和 19.8%。
正文
Abstract:In settings where labeled verifiable training data is the binding constraint, each checked example should be allocated to the model and reward density where it is most informative. We identify a reward-density principle: sparse sequence-level reward is most useful on models that can explore and discover better behavior, while dense token-level teacher supervision is better suited for compressing that behavior into a smaller deployment model; accordingly, scarce labeled data should be used upstream on the strongest available teacher, then transferred downstream as dense supervision. We evaluate this rule through a four-stage workflow:teacher RL, forward-KL warmup, on-policy distillation, optional post-bridge student RL on verifiable math and code generation with Qwen3, Llama, and Qwen2.5 models. At fixed Qwen3-1.7B deployment-student size, an RL-improved 8B teacher distilled through the dense bridge outperforms direct GRPO on the same student ($79.3\%$ vs.\ $75.9\%$ on MATH; $25.2\%$ vs.\ $19.8\%$ on AIME~2024), while transfer from the same teacher before RL underperforms. A component ablation confirms that each stage is load-bearing: the RL-improved teacher adds $7.8$ absolute points on MATH over a raw teacher of the same size, the forward-KL warmup adds $1.7$, and on-policy distillation adds $3.3$. The teacher-quality ordering: raw-teacher transfer $<$ direct GRPO $<$ RL-teacher transfer, replicates on Llama-3.1-8B-Instruct with a Llama-3.3-70B-Instruct teacher and on Qwen2.5-7B-Instruct for code generation with a Qwen2.5-32B-Instruct teacher. The operational lesson is to avoid spending scarce labeled data on the least prepared policy: use sparse reward for teacher-side discovery, dense transfer for student compression, and student-side sparse reward only after the bridge. The OPD implementation used in our experiments is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.12483 [cs.LG] |
| (or arXiv:2605.12483v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12483 arXiv-issued DOI via DataCite |
Submission history
From: Yuanda Xu [view email]
[v1]
Tue, 12 May 2026 17:57:48 UTC (52 KB)
[v2]
Thu, 14 May 2026 15:02:43 UTC (54 KB)
[v3]
Fri, 15 May 2026 15:36:53 UTC (60 KB)
[v4]
Wed, 20 May 2026 14:15:58 UTC (61 KB)
[v5]
Mon, 12 Oct 2026 03:37:41 UTC (99 KB)
来源：arXiv：cs.AI · arxiv.org