arXiv：cs.LG· Hejian Sang, Zhengze Zhou, Shayan Mohajer Hamidi, Xiaomin Li, Rohit Jain, Alborz Geramifard·· 4 小时前AI 评分32
Δ-MOPD：通过教师相对偏移实现多教师在线策略蒸馏
Composing What Each Teacher Learned: Multi-Teacher On-Policy Distillation through Teacher-Relative Shifts
AI 导读
研究者提出 Δ-MOPD，将每位教师的"教师减基座"logit 偏移重新锚定在学生冻结初始化上进行迁移，而非直接迁移教师端点策略。三个教师组合时，Δ-MOPD 较端点组合在 Math 上高出 4.11 分、五个基准上高出 1.95 分；两个教师时精度持平。分阶段路由下平均性能更高，顺序差距从 10.50 分降至 6.42 分。
正文
Abstract:Multi-teacher on-policy distillation (MOPD) is used in two settings. In common-domain composition, several teachers score each student rollout from one prompt domain and their signals form a single target; in routed-domain distillation, prompts from different domains are assigned to the corresponding specialist. Both settings usually transfer each teacher's endpoint policy, which mixes what post-training changed with preferences inherited from the teacher's base. We introduce $\Delta$-MOPD, which transfers each teacher's teacher-minus-base logit shift re-anchored at the student's frozen initialization, and compare it with endpoint supervision in both settings while holding teacher selection fixed. We first expose the mechanism that impedes endpoint transfer: inherited base pull can exceed the post-training shift. Removing it reduces the teacher-term norm ratio and target--student KL.
Across our experiments, the results suggest that shift targets are particularly useful when teacher signals are combined at a state. With three composed teachers, $\Delta$-MOPD exceeds endpoint composition by $4.11$ Math and $1.95$ five-benchmark points; with two, it matches endpoint accuracy. Under phased routing, it achieves higher mean performance in both phase orders and reduces the observed order gap from $10.50$ to $6.42$ points. Under interleaved routing, where each update involves one teacher, the two targets perform comparably. The phased results provide supporting evidence that the benefit may extend to signals accumulated across training phases. Target construction is thus an independent design axis in MOPD, complementary to teacher selection.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.10460 [cs.LG] |
| (or arXiv:2610.10460v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10460 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hejian Sang [view email]
[v1]
Wed, 7 Oct 2026 17:27:46 UTC (83 KB)
来源：arXiv：cs.LG · arxiv.org