HuggingFace Daily Papers·· 1 天前
CriticOPD：用批判引导的在线策略蒸馏恢复可靠监督
Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
AI 导读
针对在线策略蒸馏（OPD）在错误 rollout 上失效的问题，研究者提出 CriticOPD：让教师模型先批判学生的错误解答，再让学生据此重写，教师以含批判的特权信息对结果打分，而学生不带批判地进行训练。在两个 Qwen3 学生模型和六个数学推理基准上，CriticOPD 均优于 OPD，在竞赛题上提升最大。论文共 21 页，代码已公开。
正文
Abstract:On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. The resulting output, including the critique as privileged information, is scored by the teacher while the student uses the revision without the critique for training. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
| Comments: | 21 pages. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20572 [cs.LG] |
| (or arXiv:2610.20572v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20572 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kaixiang Zhao [view email]
[v1]
Wed, 14 Oct 2026 17:54:34 UTC (950 KB)
来源：HuggingFace Daily Papers · arxiv.org