arXiv：cs.LG· Narayanan Sadagopan, Yilan Chen, Vijay Lingam·· 3 小时前
基于局部信赖域优化的在策略蒸馏 Token 效用
Token Utility for On-Policy Distillation via Local Trust-Region Optimization
AI 导读
研究者提出从蒸馏目标本身推导 token 重要性，定义为在等量局部 KL 预算下每个位置损失的下降幅度，得到效用公式 s_t=g_t^⊤F_t^†g_t（g_t 为 logit 梯度，F_t 为类别 Fisher 矩阵），reverse-KL 下等价于学生加权的师生对数概率比方差，forward-KL 下等价于 Pearson χ² 散度。
正文
Abstract:Token-selection methods for on-policy distillation (OPD) prioritize student-visited positions using diverse signals, including uncertainty, teacher-student disagreement, and mismatch direction. We instead derive token importance from the distillation objective itself: how much can each position's loss decrease under an equal local KL budget on the student's predictive distribution? The answer is the utility $s_t=g_t^\top F_t^\dagger g_t$, where $g_t$ is the logit gradient and $F_t$ is the categorical Fisher matrix. For reverse-KL it equals the student-weighted variance of the student-teacher log-probability ratio, and for forward-KL it equals the Pearson $\chi^2$ divergence, so the appropriate utility follows from the choice of objective. On shared training checkpoints, several independently motivated criteria, including student entropy, KL disagreement, and TIP, concentrate their importance on nearly the same positions as the derived utility at a 50\% selection fraction, while diverging at stricter fractions and for direction-specific criteria. We instantiate the utility as Fisher Utility Weighting (FUW), a continuous, mean-normalized token weighting. Across seven mathematical reasoning benchmarks and three Qwen3 student-teacher pairs, FUW improves over matched uniform OPD under both reverse-KL (1.35-4.0 macro points) and forward-KL (1.08-2.71 macro points), and is competitive with existing selective methods using a single closed-form score derived from the training objective.
| Comments: | 10 pages, Accepted to Pre-to-Post workshop at NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18883 [cs.LG] |
| (or arXiv:2610.18883v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18883 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Narayanan Sadagopan [view email]
[v1]
Thu, 1 Oct 2026 20:01:20 UTC (1,992 KB)
来源：arXiv：cs.LG · arxiv.org