arXiv：cs.AI· Han Cui, Jianhao Yan, Yun Luo, Hongbo Zhang, Zhizhang Fu, Yue Zhang·· 5 小时前AI 评分45
从强化学习视角看 On-Policy Distillation 的收益与崩溃
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
AI 导读
研究从强化学习视角解释 On-Policy Distillation（OPD）为何既提升性能又可能崩溃为冗长重复生成：教师模型隐式奖励学生行为，即使自身很少产生这类文本。实验显示 OPD 并未扩展学生能力，只是让正确回答更易被采样；当隐式奖励与质量错位时会出现 reward hacking，放大冗长重复的 rollout。训练中屏蔽不健康回答或使用 SFT 初始化均可缓解崩溃。
正文
Abstract:On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.03185 [cs.AI] |
| (or arXiv:2610.03185v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.03185 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Han Cui [view email]
[v1]
Fri, 2 Oct 2026 11:57:36 UTC (2,399 KB)
来源：arXiv：cs.AI · arxiv.org