arXiv：cs.CL· Lei Zhao, Qichao Zhao, Bowen Zuo, Qishi Zhan·· 3 小时前
为什么 On-Policy Distillation 有时会失败：学习信号消失
Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals
AI 导读
研究揭示 on-policy distillation（OPD）使用大规模教师模型时会出现早期损失平台期：在代码生成和数学推理任务中，经过 200 次更新后平均最终损失仅降低 25.1%，而 self-RL 教师可达 96.2%。
正文
Abstract:On-policy distillation (OPD) enables effective capability transfer between language models, yet the mechanisms underlying its failures are not fully understood. Across code generation and mathematical reasoning, OPD with larger-scale teachers exhibits early loss plateaus, with an average final loss reduction of 25.1% after 200 updates, compared with 96.2% for self-RL teachers, obtained by further reinforcement learning (RL) training of the initial student. To understand this difference, we analyze OPD as an idealized continuous-time dynamical system in the small-learning-rate limit. Our training-log diagnostics associate these plateaus with an early decline in a gradient-based learning-signal proxy while substantial loss remains; these measurements do not establish why the underlying gradient weakens. We further prove a local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions, offering a conditional explanation for the success of self-RL teachers in our experiments. Across runs with and without loss plateaus, we observe small relative parameter changes (0.025-0.098%) and high similarity between the student's representations before and after OPD (linear CKA $>0.98$ across layers). These observations suggest that limited representation adaptation may contribute to learning-signal collapse, a hypothesis that remains to be tested. Code is available at this https URL.
| Comments: | 50 pages. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.11247 [cs.LG] |
| (or arXiv:2610.11247v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11247 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lei Zhao [view email]
[v1]
Thu, 8 Oct 2026 04:52:05 UTC (592 KB)
来源：arXiv：cs.CL · arxiv.org