arXiv：cs.AI· Haowei Guo, Baolong Bi, Ruicheng Zhang, Cong Li, Bingqian Sun, Zhonghai Wu·· 13 小时前
教师模型何时该更新？自同策略蒸馏中的时序耦合与稳定性研究
When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation
AI 导读
在 Qwen3-8B 上的受控调度扫描发现，教师模型更新之间的完全冻结期（isolation periods）而非教师模型新旧程度，才是稳定学习的关键。研究提出时序 KL 结构诊断框架，揭示短视最优固定调度在长时训练中会因时钟驱动刷新导致"状态无视崩溃"。为此提出的 CGTR 在 Chemistry、Biology、Physics、ToolUse 四项任务上实现零崩溃和最佳最终得分。
正文
Abstract:Self on-policy distillation trains a student policy against a teacher derived from its own parameter history, yet the teacher's update schedule -- which governs the \emph{temporal coupling} between teacher and student -- has not been systematically studied as a stability variable. Through a controlled schedule sweep on Qwen3-8B, we establish that \emph{isolation periods}, defined as complete teacher freezing between updates, are the key structural property enabling stable learning, not teacher age. To characterize these underlying training dynamics, we introduce a diagnostic framework of temporal KL structure, refresh shock, and length-tail risk. This framework further uncovers \emph{state-oblivious collapse}: optimal short-horizon fixed schedules catastrophically fail under long-horizon training because a clock-driven refresh can copy a transiently drifting student into the teacher in a single, irreversible step. This failure mode is invisible under short-horizon evaluation and mechanistically distinct from EMA's chronic contamination. To address this, we propose \emph{Consolidation-Gated Teacher Refresh} (CGTR), which preserves isolation periods while gating each refresh on joint evidence of reward improvement and length-tail safety, ensuring every teacher movement responds to genuine student consolidation rather than a clock signal. With a single shared parameter set and no per-dataset retuning, CGTR achieves \textbf{zero collapse} and the best final score on all four tasks (Chemistry, Biology, Physics, ToolUse), self-regulating its refresh frequency to each task's learning dynamics.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.03532 [cs.LG] |
| (or arXiv:2606.03532v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.03532 arXiv-issued DOI via DataCite |
Submission history
From: Haowei Guo [view email]
[v1]
Tue, 2 Jun 2026 11:54:39 UTC (1,233 KB)
[v2]
Fri, 9 Oct 2026 13:22:21 UTC (1,103 KB)
来源：arXiv：cs.AI · arxiv.org