arXiv：cs.LG· Kutay Tire, Atula Tejaswi, Haris Vikalo, Sujay Sanghavi·· 4 小时前
INTACT：在冻结骨干模型上诱导潜在推理
INTACT: Inducing Latent Reasoning on a Frozen Backbone
AI 导读
INTACT 用一个小型外部编码器在冻结骨干模型上诱导潜在推理，将隐藏状态迭代映射为潜在推理前缀，再由冻结模型输出文本推理与答案。
正文
Abstract:Latent reasoning replaces natural-language chains of thought with compact continuous "soft-token" representations, reducing the number of intermediate tokens that must be decoded autoregressively. We use a small external encoder to induce latent reasoning in a frozen backbone model. The encoder iteratively maps the backbone's hidden states to new latent blocks, constructing a latent reasoning prefix, after which the frozen model writes its remaining reasoning and its answer as text. Our method, INTACT, trains this encoder through distillation to reproduce the effect of written reasoning steps while keeping the model's behavior close to that of the frozen backbone, and then reinforces it on answer correctness. Across four frozen backbones from 1B to 8B, INTACT improves both GSM8K and average out-of-domain accuracy, including GSM8K gains from 75.8% to 79.3% on Llama-3.2-3B and from 77.6% to 79.7% on Qwen2.5-3B. It also reduces end-to-end latency per problem by 19% on Llama-3.2-3B and outperforms the evaluated latent-reasoning baselines at the 3B and 8B scales. Beyond the mathematics setting, the same recipe trained on a multi-domain corpus improves the frozen backbone on twelve evaluation benchmarks spanning mathematics, science, and commonsense, without task-specific adaptation.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19568 [cs.CL] |
| (or arXiv:2610.19568v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19568 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kutay Tire [view email]
[v1]
Wed, 14 Oct 2026 04:35:34 UTC (251 KB)
来源：arXiv：cs.LG · arxiv.org