arXiv：cs.AI· Yuhang Yang, Mingxuan Xia, Chao Ye, Shuai Zhu, Shouqing Yang, Haobo Wang, Shenzhi Yang, Zhengqing Zang, Chenglong Wang·· 4 小时前
PreSO：面向同策略自蒸馏的特权表征隔离方法
PreSO: Privileged Representation Isolation for On-Policy Self-Distillation
AI 导读
针对同策略自蒸馏（OPSD）中教师优势依赖学生推理时不可得信息的问题，研究者提出特权表征隔离方法 PreSO，用紧凑提示限制解细节进入特权偏移，并从跨问题池化的偏移中估计低秩子空间、将特权修正投影后加回学生表征作为蒸馏目标。在五个竞赛级数学基准上，PreSO 于 Qwen3-1.7B、4B 和 8B 上平均准确率较基座模型分别提升 6.8、9.0 和 10.0 分。
正文
Abstract:On-policy self-distillation (OPSD) uses a self-teacher conditioned on privileged information to turn the sparse outcome signal of reinforcement learning with verifiable rewards (RLVR) into dense token-level supervision. Yet the teacher's advantage may depend on information unavailable to the student at inference, raising a central question: which privileged changes are worth distilling? We use cross-problem recurrence as an empirical cue for potentially reusable guidance. Comparing the representation shifts induced by full solutions and compact hints, we find that hint-induced shifts are more concentrated and directionally consistent across problems, and yield more sustained downstream gains despite revealing less information. Motivated by this association, we propose Privileged Representation Isolation for On-Policy Self-Distillation (PreSO). At the source, PreSO uses compact hints to limit the solution-specific detail entering the privileged shift. At the target, it estimates a leading low-rank subspace from shifts pooled across related problems, projects the privilege-induced correction onto this subspace, and adds it to the student's unprivileged representation to form the representation-level distillation target. The selected subspace serves as an operational proxy for recurring structure, rather than a guaranteed separation of transferable and instance-specific information. On five competition-level mathematics benchmarks, PreSO consistently outperforms strong baselines across Qwen3-1.7B, 4B, and 8B, improving average accuracy over the base model by 6.8, 9.0, and 10.0 points. Beyond peak accuracy, PreSO attains the highest final-checkpoint average at every model scale, with improvements that persist through the end of training.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14001 [cs.LG] |
| (or arXiv:2610.14001v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14001 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhang Yang [view email]
[v1]
Sun, 27 Sep 2026 08:48:42 UTC (1,446 KB)
来源：arXiv：cs.AI · arxiv.org