arXiv：cs.LG· Fei Wu, Zhenrong Zhang, Jianshu Zhang, Jun Du·· 6 小时前
VASPO：结合特权价值估计的自蒸馏方法
VASPO: Self-Distillation with Privileged Value Estimation
AI 导读
研究者提出 VASPO（Value-Augmented Self-Distillation Policy Optimization），将特权信息自蒸馏与 PPO 结合：自教师与 critic 在学生生成的前缀上评估相同的特权信息，critic 计算 token 级 PPO 优势，自教师用有界概率比调节选定的正向优势。
正文
Abstract:On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and the teacher for signal scaling. However, we demonstrate a more effective integration within PPO. Specifically, we identify complementary positional dynamics: privileged information enhances a critic's value-prediction accuracy toward the response's end, whereas self-teacher guidance is strong initially, drops mid-generation, and rebounds finally. These insights motivate Value-Augmented Self-Distillation Policy Optimization (VASPO), wherein a self-teacher and critic evaluate identical privileged information at student-generated prefixes. The critic computes token-level PPO advantages from predicted continuation returns, while the self-teacher modulates selected positive advantages via bounded probability ratios. Across six mathematical reasoning benchmarks, VASPO improves mean accuracy over PPO by 2.59 and 1.81 percentage points on Qwen3-4B and OLMo3-7B variants, respectively, and achieves the highest mean accuracy among evaluated methods on Knights and Knaves. Our code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20516 [cs.CL] |
| (or arXiv:2610.20516v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20516 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fei Wu [view email]
[v1]
Wed, 14 Oct 2026 17:17:18 UTC (6,247 KB)
来源：arXiv：cs.LG · arxiv.org