arXiv：cs.LG· Nhat Minh Pham, Duy Tung Doan, Quang Huy Phung, Thi Duyen Ngo, Vinh Van Nguyen, Khac-Hoai Nam Bui·· 3 小时前
SPARK：在推测式 GRPO rollout 中从验证反馈中学习
SPARK: Learning from Verification in Speculative GRPO Rollouts
AI 导读
SPARK 是一种轻量自适应机制，利用目标模型的验证反馈在生成过程中修正 draft 模型预测，且不改变其原有训练流程。在 21 个模型-数据集设置上，SPARK 相比 EAGLE3 风格与 DFlash 风格基线，生成加速分别提升 32.7% 和 11.4%，GRPO 端到端训练加速分别提升 32.2% 和 8.5%。
正文
Abstract:Group Relative Policy Optimization (GRPO) has become a standard approach for improving the reasoning abilities of large language models, but its training is costly because each step requires generating many long reasoning responses. Recent methods use speculative decoding to accelerate the rollouts, where a draft model proposes tokens that the target model verifies. However, we find that the draft model repeatedly misses the same tokens within a response and across responses in the same GRPO group, and the verification feedback on these misses is not used to update the draft during the rollout. Motivated by this analysis, we introduce SPARK, a lightweight self-adaptive mechanism designed to enhance draft models used in speculative GRPO training. SPARK leverages target verification feedback to refine draft predictions during generation while preserving their existing training procedures. Averaged across 21 model-dataset settings, SPARK improves generation speedup by 32.7% and 11.4%, and end-to-end GRPO training speedup by 32.2% and 8.5%, over EAGLE3-style and DFlash-style baselines, respectively.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.04962 [cs.LG] |
| (or arXiv:2608.04962v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04962 arXiv-issued DOI via DataCite |
Submission history
From: Tung Doan [view email]
[v1]
Wed, 5 Aug 2026 15:32:27 UTC (1,159 KB)
[v2]
Wed, 14 Oct 2026 04:53:03 UTC (311 KB)
来源：arXiv：cs.LG · arxiv.org