arXiv：cs.AI· Taehyeon Kim, Hyunsoo Lee, Youngsoo Jang, Moontae Lee·· 22 小时前
RPO：通过根 token 策略优化让大推理模型自适应思考
Efficiently Learning To Reason or Not to Reason: Root-token Policy Optimization for Adaptive Thinking
AI 导读
针对大推理模型对难度不同的问题一律重度计算导致的过度思考问题，研究者提出 RPO（Root-token Policy Optimization），仅训练初始根 token（如是否调用相应标签）来自主决定何时推理，从而大幅降低训练开销与显存占用。在多个模型族和规模上，RPO 以约为此前自适应推理方法 2% 的训练算力学会了难度感知的自适应思考。
正文
Abstract:Large reasoning models (LRMs) achieve strong performance by externalizing explicit reasoning traces before producing the answer, yet suffer from overthinking challenge that allocates uniformly heavy computation to queries of varying difficulty. While proprietary models mitigate this via opaque routing, open-source LRMs still lack an efficient mechanism to internalize adaptive reasoning due to both expensive training cost and limited disclosure of training recipes. In response, we introduce RPO (Root-token Policy Optimization), a framework that enables LRMs to self-determine when to reason by training only the initial root token (e.g., whether to invoke the <think> tag) via group relative reward and group-wise advantages. By focusing on this pivotal branching point, RPO drastically reduces training overhead and VRAM usage. Across multiple model families and scales, RPO learns difficulty-aware adaptive thinking at just ~2% of the training compute of prior adaptive-reasoning methods.
| Comments: | ACL 2026 Oral |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17767 [cs.CL] |
| (or arXiv:2610.17767v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17767 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Taehyeon Kim [view email]
[v1]
Tue, 13 Oct 2026 05:53:34 UTC (474 KB)
来源：arXiv：cs.AI · arxiv.org