arXiv：cs.LG· Hao Li, MeiJia Chen, Weijie Ren, Donghan Li, Zijun Tian, Jingchun Huang, Naibo Wang·· 7 小时前
RIDE：在策略蒸馏中直接外推 RL 诱导的表示残差
The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
AI 导读
研究者提出 RIDE（RL-Induced Direction Extrapolation），将强化学习相对于基座检查点的内部表示偏移方向作为外推目标，在每一层和每个 token 位置把学生隐藏状态回归到沿该残差方向超越教师的位置。该方法在覆盖不同规模、架构和预训练来源的四个基座/RL 教师组合上均接近或超过 RL 训练的教师，且是唯一均值达标的方法，并持续优于输出空间外推。
正文
Abstract:On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The language-model head, however, attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at a small fraction of its weight, and the sampled-token log-probability ratios on which output-space extrapolation relies inject noise that the extrapolation amplifies, making training unstable. We observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Motivated by this observation, we propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space: at every layer and token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual. Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher, which makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher. Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so, and it consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. Project page: this https URL.
| Comments: | 19 pages; revised related-work discussion |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.36484 [cs.LG] |
| (or arXiv:2609.36484v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.36484 arXiv-issued DOI via DataCite |
Submission history
From: Hao Li [view email]
[v1]
Tue, 29 Sep 2026 01:44:39 UTC (641 KB)
[v2]
Wed, 14 Oct 2026 09:17:52 UTC (641 KB)
来源：arXiv：cs.LG · arxiv.org