arXiv：cs.CL· Zhiyu Cao, Kaixin Wu, Mingjie Zhong, Peifeng Li, Can Ye, Qiaoming Zhu·· 4 小时前AI 评分31
HDPO：面向 LLM 推理的提示引导多样化策略优化
Hint-Guided Diversified Policy Optimization for LLM Reasoning
AI 导读
研究者提出 Hint-Guided Diversified Policy Optimization（HDPO），让 LLM 先列出所有候选解题思路作为提示，再选出最可靠的一条继续推理。该方法包含结构化推理冷启动与提示引导多样化强化学习两个阶段，遵循“propose-select-think”轨迹。实验显示 HDPO 有效提升 LLM 推理能力，并增强候选解的多样性及模型识别可靠解的能力。
正文
Abstract:Recent developments in Large Language Models (LLMs) have showcased impressive reasoning capabilities, with Reinforcement Learning with Verifiable Rewards (RLVR) being a promising enhancement strategy. However, existing reward mechanisms are constrained to the outcome-level correctness and lack explicit signals to guide the model to consider diverse solutions. In contrast, human problem solving typically involves evaluating multiple potential approaches and selecting the most reliable solution, a cognitive process that current RLVR frameworks do not explicitly incentivize. Inspired by this, we propose Hint-Guided Diversified Policy Optimization (HDPO), allowing the model to first list all potential candidate solution outlines as hints and then select the most reliable one for further reasoning. HDPO comprises two stages of Cold Start for Structured Reasoning and Hint-Guided Diversified Reinforcement Learning to incentivize the model to generate diverse and reliable solutions following the ``propose-select-think'' trajectory. Experimental results show that HDPO effectively boosts LLM reasoning and enhances the diversity of candidate solutions as well as the LLM's ability to identify reliable solutions.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.03021 [cs.CL] |
| (or arXiv:2606.03021v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.03021 arXiv-issued DOI via DataCite |
Submission history
From: Zhiyu Cao [view email]
[v1]
Tue, 2 Jun 2026 01:55:54 UTC (2,206 KB)
[v2]
Fri, 24 Jul 2026 07:13:48 UTC (2,206 KB)
[v3]
Mon, 27 Jul 2026 12:24:24 UTC (2,205 KB)
[v4]
Fri, 2 Oct 2026 07:02:33 UTC (2,198 KB)
来源：arXiv：cs.CL · arxiv.org