arXiv：cs.LG· Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang, Geng Liu, Haiyang Guo, Guo-Sen Xie, Gaofeng Meng, Hongbin Liu, Fei Zhu·· 5 小时前AI 评分48
密度不等于更好：持续后训练中在线自蒸馏的局限
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
AI 导读
研究通过自蒸馏策略优化（SDPO）重新审视在线学习可缓解遗忘的乐观主张：SDPO 在教师信号稳定对齐时能加速域内专精，但难以泛化到分布外。在持续后训练中，SDPO 遗忘更严重甚至崩溃，而 GRPO 适应更保守、更好保留原有能力。分析将其归因于参数与响应空间漂移加剧，以及师生自强化循环放大高频伪影，说明仅靠在线数据不足以支撑持续学习。
正文
Abstract:Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with self-distillation as a particularly attractive approach. We revisit this optimistic claim through self-distillation policy optimization (SDPO). Our experiments show that SDPO accelerates in-domain specialization when teacher signals are stable and well aligned, but struggles to generalize out of distribution. In continual post-training, SDPO exhibits greater forgetting and can even collapse, whereas GRPO, the more established on-policy reinforcement learning method, adapts more conservatively and better preserves prior capabilities. Further analyses link these failures to increased drift in parameter and response space, and to amplification of high-frequency artifacts through a self-reinforcing teacher-student loop. Thus, on-policy data alone is insufficient for continual learning. Self-distillation is effective when teacher targets are stable and token-level supervision is reliable, but should not be treated as a default stabilizer for continual post-training. Our code is available at this https URL.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.01763 [cs.LG] |
| (or arXiv:2607.01763v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01763 arXiv-issued DOI via DataCite |
Submission history
From: Meng Wang [view email]
[v1]
Thu, 2 Jul 2026 06:24:30 UTC (881 KB)
[v2]
Fri, 2 Oct 2026 07:51:19 UTC (876 KB)
来源：arXiv：cs.LG · arxiv.org