arXiv：cs.AI· Yuran Wang, Zekun Wang, Bohan Zeng, Ruixu Zhang, Wenxuan Liu, Liu Yang, Yifan Dai, Yang Shi, Bozhou Li, Chengzhuo Tong, Daili Hua, Yuanxing Zhang, Wentao Zhang·· 6 小时前
Flux-OPD：用演化上下文做同策略蒸馏
Flux-OPD: On-Policy Distillation with Evolving Contexts
AI 导读
Flux-OPD 提出一种同策略蒸馏（OPD）范式，把任务反馈衍生的演化上下文用作训练中监督。该方法通过 reverse-KL 分解刻画上下文蒸馏，将上下文条件教师与无上下文教师的差异作为上下文修正注入，并按冲突项加权修正强度。实验显示 Flux-OPD 在各类 OPD 范式中表现最佳。
正文
Authors:Yuran Wang, Zekun Wang, Bohan Zeng, Ruixu Zhang, Wenxuan Liu, Liu Yang, Yifan Dai, Yang Shi, Bozhou Li, Chengzhuo Tong, Daili Hua, Yuanxing Zhang, Wentao Zhang
Abstract:Large language model training in open-ended domains lacks verifiable rewards, making task preferences difficult to formalize as effective supervision. Contexts derived from task feedback can provide richer supervision, yet fixed contexts cannot adapt as the student evolves, motivating contexts that evolve with student performance. However, directly using evolving contexts as in-training supervision results in unstable distillation targets and conflicting distributions, requiring mechanisms to stabilize the target and downweight conflicts. In this paper, we characterize context distillation through a reverse-KL decomposition, showing that the student is distilled toward the geometric mean of context-conditioned teachers, while the objective contains a conflict term measuring disagreement among these teachers. Motivated by this characterization, we propose Flux-OPD, an OPD paradigm that uses evolving contexts as in-training supervision to incorporate contextual signals derived from task feedback. Flux-OPD treats differences between context-conditioned and context-free teachers as contextual difference signals, injects them as contextual corrections into the context-free teacher, and weights the correction strength using the conflict term. Experiments show that Flux-OPD achieves the best results among OPD paradigms, highlighting the potential of combining teacher supervision with evolving contexts as in-training supervision.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.28022 [cs.LG] |
| (or arXiv:2607.28022v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.28022 arXiv-issued DOI via DataCite |
Submission history
From: Yuran Wang [view email]
[v1]
Thu, 30 Jul 2026 11:11:32 UTC (2,758 KB)
[v2]
Mon, 12 Oct 2026 06:47:25 UTC (2,769 KB)
来源：arXiv：cs.AI · arxiv.org