arXiv：cs.AI· Yingdong Shi, Zhihong Li, Changming Li, Jingyi Yu, Quan Li, Kan Ren·· 17 小时前
OPMD：超越输出监督的在线策略机制蒸馏
OPMD: On-Policy Mechanism Distillation Beyond Output Supervision
AI 导读
研究者提出 On-Policy Mechanism Distillation（OPMD），在在线策略蒸馏（OPD）的输出监督之外，进一步迁移教师模型的中间计算过程。OPMD 先构建与学生兼容的教师模型，将教师表示转换为学生下游计算可用的中间状态，再从在线策略轨迹中内化这些状态。在三种教师-学生设定中，OPMD 的平均准确率均高于所评估的蒸馏基线，并可泛化到多教师蒸馏与跨家族监督微调。
正文
Abstract:On-policy distillation (OPD) improves language-model by supervising a student on prefixes sampled from its own policy, but its supervision remains confined to the output space. Consequently, OPD transfers what a teacher predicts without supervising the intermediate computation that produces those predictions. Simply distilling intermediate representations like representation matching is insufficient since teacher and student representations may differ geometrically, and even well-aligned hidden states need not be usable by the student's subsequent computation. We introduce On-Policy Mechanism Distillation (OPMD), which transfers teacher-derived intermediate computation while explicitly accounting for its compatibility with the student. OPMD first constructs a student-compatible teacher that transforms teacher representations into intermediate states usable by the student's downstream computation, then it internalizes these states from on-policy trajectories. OPMD achieves the highest average accuracy among the evaluated distillation baselines in all three teacher--student settings, and generalizes to multi-teacher distillation and cross-family supervised fine-tuning. Further analyses show that mechanism supervision yields more effective reasoning and meaningfully alters the student's intermediate computation and solution strategies, supporting that it transfers information beyond output supervision alone. Together, these results establish mechanism-level supervision as an effective way to move on-policy distillation beyond output supervision, enabling students to learn inner mechanisms they can use and internalize.
| Comments: | 21 pages, 6 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14022 [cs.LG] |
| (or arXiv:2610.14022v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14022 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yingdong Shi [view email]
[v1]
Tue, 29 Sep 2026 14:07:18 UTC (615 KB)
来源：arXiv：cs.AI · arxiv.org