用分布对齐提示合成与反向提示退火缓解数学 RLVR 中的分布锐化

View PDF HTML (experimental)

Abstract:In reinforcement learning with verifiable rewards (RLVR), challenging math questions whose sampled groups remain uniformly incorrect yield no informative update, letting easier questions dominate actor updates. This imbalance can contribute to distribution sharpening: low-$k$ accuracy improves while problem coverage contracts. Hint scaffolding can restore informative updates by revealing a prefix of a verified teacher solution. However, existing hint-based approaches still face two issues: teacher--student distribution mismatch and the need to reduce hint exposure to match no-hint evaluation. We address these issues through two components. Distribution-Aligned Hint Synthesis (DAHS) constructs verified teacher hints conditioned on student-style responses. Backward Hint Annealing (BHA) anneals hint exposure across length buckets as bucket accuracy improves. BHA also uses per-question hint dropout to keep no-hint rollouts throughout RL training. Under the DAPO training framework, our method improves both pass@1 and pass@2048 over DAPO on AIME24/25/26 for Qwen3-1.7B-Base and Qwen3-4B-Base, and improves pass@1 on five secondary math benchmarks (Olympiad, MATH500, Minerva Math, AMC23, and GSM8K). On Llama-3.2-1B-Instruct, the gains concentrate in the high-$k$ regime without recovering the base model's coverage. Overall, when verified teacher prefixes are available, DAHS+BHA mitigates DAPO-induced contraction of high-$k$ problem coverage in the evaluated challenging-math RLVR settings.

Subjects:	Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as:	arXiv:2604.07747 [cs.AI]
	(or arXiv:2604.07747v2 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2604.07747 arXiv-issued DOI via DataCite

Submission history

From: Pei-Xi Xie [view email]
[v1] Thu, 9 Apr 2026 03:08:41 UTC (806 KB)
[v2] Mon, 12 Oct 2026 08:25:32 UTC (1,127 KB)