arXiv：cs.AI· Shuo Yang, Zheyu Zhang, Mahito Sugiyama, Gjergji Kasneci·· 19 小时前
当训练集中于失败题时，On-Policy Distillation 为何反而让学生模型退步？
Correcting or Continuing? What On-Policy Distillation Learns from Student Errors
AI 导读
研究用 Qwen-3 和 Phi-4 在统一竞赛数学协议下评测 5 种代表性 OPD 机制，发现当训练只集中在学生当前做错的题目上时，OPD 无法提升准确率，反而让全部 8 种配置性能下降（双尾符号检验 p=0.0078），尽管训练 KL 散度持续降低。相同流程在不集中于失败题时准确率提升 4.6 个百分点，说明这一退步取决于训练数据构成而非方法本身。
正文
Abstract:On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher, mitigating the distribution mismatch of off-policy supervision. While prior work has primarily trained on problems a student can largely solve, we ask whether OPD can improve a student when training is restricted to problems it currently fails. We evaluate five representative OPD mechanisms with Qwen-3 and Phi-4 models under a unified competition-mathematics protocol. We find that OPD does not improve accuracy on this hard subset, with most mechanisms instead degrading performance despite continued decreases in training KL divergence. All eight configurations show a decline, and the two-sided sign test gives p=0.0078. In contrast, the same pipeline improves accuracy by 4.6 percentage points when training is not concentrated on failures, showing that the failure is conditional on the training regime rather than systematic. These findings suggest that OPD is effective when supervision is dominated by correct student prefixes, but can reinforce existing errors when training is concentrated on problems the student currently fails.
| Comments: | 10 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18011 [cs.LG] |
| (or arXiv:2610.18011v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18011 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuo Yang [view email]
[v1]
Tue, 13 Oct 2026 09:10:58 UTC (318 KB)
来源：arXiv：cs.AI · arxiv.org