arXiv：cs.LG（机器学习，全量分类）· Langlin Huang, Hao Liu, Mononito Goswami, Xinyu Li, Prithwith Jana, Nikos Kanakaris, Patrick Bl\"obaum, Purak Jain·· 5 小时前AI 评分32
SCOUT：用学生条件化教师更新解决在线蒸馏中的离策略教师问题
On the Off-Policy Teacher in On-Policy Distillation
AI 导读
针对在线策略蒸馏（OPD）中教师模型面对学生生成前缀时表现随前缀变长而下降的离策略问题，研究者提出协同训练框架 SCOUT，通过带可验证奖励的强化学习定期优化教师从学生前缀继续生成的能力。在多种教师—学生配置、模型规模和推理领域上，SCOUT 均持续提升了在线策略蒸馏的效果。
正文
Abstract:On-policy distillation (OPD) has recently emerged as a promising post-training paradigm in which the student learns from trajectories generated by its own policy under dense teacher supervision. However, OPD introduces a fundamental asymmetry: although the sampled trajectories are on-policy for the student, they are off-policy for the teacher. The teacher is typically optimized to continue from prefixes generated by its own policy, but during OPD it must instead supervise prefixes generated by the student. Empirically, we find that its continuation performance degrades as these prefixes grow longer. To address this issue, we propose Student-COnditioned Updates of the Teacher (SCOUT), a co-training framework that adapts the teacher to student-generated prefixes. Alongside standard OPD updates, SCOUT periodically optimizes the teacher's conditional ability using reinforcement learning with verifiable rewards, where the teacher generates continuations from student prefixes and learns from outcome rewards. Controlled experiments show that SCOUT improves the teacher's ability to continue from student-generated prefixes, supporting the intended mechanism of student-conditioned teacher adaptation. Across multiple teacher--student configurations, model scales, and reasoning domains, SCOUT also consistently improves the effectiveness of on-policy distillation.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.38360 [cs.LG] |
| (or arXiv:2609.38360v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38360 arXiv-issued DOI via DataCite |
Submission history
From: Langlin Huang [view email]
[v1]
Tue, 29 Sep 2026 18:22:05 UTC (683 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org