arXiv：cs.LG（机器学习，全量分类）· Linjian Meng, Siyuan Gan, YuHan Li, Xiran Wang, Ziyang Ding, Ditang Gou, Yiming Wu, Zhen Zhao·· 5 小时前AI 评分33
面向在线策略蒸馏的无偏 Top-k 估计
Unbiased Top-$k$ Estimation for On-Policy Distillation
AI 导读
针对 Top-k 在线策略蒸馏（TK-OPD）因丢弃 top-k 之外概率质量而产生偏差的问题，研究者提出 Tail-Corrected Top-k On-Policy Distillation（TT-OPD），在保留丰富分布监督和低计算成本的同时，给出反向 KL 散度梯度的无偏估计。
正文
Abstract:On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning capability of a strong teacher LLM to a weaker student LLM. OPD trains the student by minimizing the reverse KL divergence between the teacher and the student via rollouts generated by the student's policy. However, estimating the gradient of the reverse KL divergence in OPD remains a challenge. Using only the sampled token from the student-generated rollout is computationally cheap but provides limited distributional supervision, which will degrade accuracy. In addition, using the full vocabulary provides complete distributional supervision but is computationally expensive. Therefore, recent works propose Top-$k$ OPD (TK-OPD) that use selected top-$k$ tokens, which provides richer distributional supervision than sampled-token estimation at substantially lower computational cost than full-vocabulary estimation. Unfortunately, using only the selected top-$k$ tokens induces bias, leading to accuracy degradation, as the probability mass outside the selected top-$k$ tokens is discarded. To address the bias of TK-OPD, we propose Tail-Corrected Top-$k$ On-Policy Distillation (TT-OPD). It preserves the advantages of TK-OPD, including rich distributional supervision and low computational cost, while providing an unbiased estimator of the gradient of the reverse KL divergence. The key insight of TT-OPD is to use not only the selected top-$k$ tokens, but also the sampled token from the student-generated rollout, thereby recovering the discarded probability mass in expectation, avoiding the bias. Experimental results demonstrate that TT-OPD significantly outperforms other tested OPD variants.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2609.34447 [cs.CL] |
| (or arXiv:2609.34447v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.34447 arXiv-issued DOI via DataCite |
Submission history
From: Linjian Meng [view email]
[v1]
Mon, 28 Sep 2026 07:03:32 UTC (221 KB)
[v2]
Thu, 1 Oct 2026 03:43:03 UTC (220 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org