SR-OPD：用成功轨迹引导的在线策略蒸馏，教师 token 消耗降至 3.46-5.02%

View PDF HTML (experimental)

Abstract:On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.

Comments:	18 pages, 2 figures. Submitted to ICLR 2027
Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.02678 [cs.AI]
	(or arXiv:2610.02678v1 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2610.02678 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Xiang Chen [view email]
[v1] Fri, 2 Oct 2026 01:59:53 UTC (455 KB)