arXiv：cs.CL· Guo Tang, Shixin Jiang, Heng Chang, Zihan Zhang, Nuo Chen, Yuhan Li, Huiming Fan, Jia Li, Ming Liu, Bing Qin·· 16 小时前
LoopRPT：面向循环语言模型的强化预训练框架
LoopRPT: Reinforcement Pre-Training for Looped Language Models
AI 导读
研究者提出 LoopRPT，一种专为循环语言模型（LoopLM）设计的强化预训练框架，将下一 token 预测重构为下一 token 推理任务，通过 EMA teacher 参考和带噪隐状态 rollout 把强化信号直接作用于隐式计算步骤。该方法在 Ouro 多个模型规模上验证，提升了单步表示质量，在准确率-计算量权衡上实现 Pareto 占优，并在困难 token 上取得大幅提升。
正文
Abstract:Looped language models (LoopLMs) perform iterative latent computation to refine internal representations, offering an alternative to explicit chain-of-thought (CoT) reasoning. However, existing reinforcement learning (RL) paradigms primarily target output tokens, creating a structural mismatch with looped architectures whose reasoning unfolds implicitly. In this work, we propose LoopRPT, a reinforcement pre-training framework tailored for LoopLMs. By reframing next-token prediction as a next-token reasoning task, LoopRPT assigns reinforcement signals directly to latent steps using an EMA teacher reference and noisy latent rollouts. This formulation enables RL to directly shape intermediate representations, compressing effective reasoning into fewer iterations. We instantiate LoopRPT on Ouro across multiple model scales. Results show that LoopRPT improves per-step representation quality and achieves Pareto dominance in accuracy-computation trade-offs. Large gains on hard tokens indicate that LoopRPT enhances early-stage reasoning rather than merely encouraging premature exits. Our findings highlight reinforcement pre-training as a principled paradigm for learning efficient latent reasoning in LoopLMs.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.19714 [cs.CL] |
| (or arXiv:2603.19714v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.19714 arXiv-issued DOI via DataCite |
Submission history
From: Shixin Jiang [view email]
[v1]
Fri, 20 Mar 2026 07:35:38 UTC (2,741 KB)
[v2]
Wed, 14 Oct 2026 06:34:29 UTC (5,986 KB)
来源：arXiv：cs.CL · arxiv.org