arXiv：cs.CL· Ryan Solgi, Jiayi Tian, Zheng Zhang·· 3 小时前
LoRi：面向隐式推理的低秩蒸馏框架
LoRi: Low-Rank Distillation for Implicit Reasoning
AI 导读
研究者提出 LoRi 低秩蒸馏框架，通过共享低秩张量子空间对齐教师与学生的推理轨迹，将推理内化到大语言模型中。在 LLaMA、Qwen 等多个模型系列的数学推理基准上，该方法持续提升性能，在多步任务上接近显式 CoT 准确率，并优于此前的 iCoT 蒸馏方法。
正文
Abstract:Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.05315 [cs.CL] |
| (or arXiv:2606.05315v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.05315 arXiv-issued DOI via DataCite |
Submission history
From: Ryan Solgi [view email]
[v1]
Wed, 3 Jun 2026 18:05:50 UTC (1,259 KB)
[v2]
Wed, 7 Oct 2026 18:39:29 UTC (1,261 KB)
来源：arXiv：cs.CL · arxiv.org