arXiv：cs.LG· Abiel Malkani Talwar, Christopher Nemeth, Henry Moss·· 13 小时前
研究质疑 LLM 测试时锐化的 Power Sampling 方法
A Critical Look at Power Sampling when Test Time Sharpening LLMs
AI 导读
针对无需奖励信号、通过 MCMC 或 SMC 从幂分布 π_α ∝ π_base^α 采样的"幂采样"方法，研究显示这些采样器在现实算力预算下无法收敛到 π_α，其收益大多可由合适的 Best-of-N 复现。直接微调基础模型以匹配 π_α 后，收益仍有噪声且仅对特定模型和采样任务有效，说明幂采样的价值在于提取基础模型本就认为高概率的序列，而非精确逼近 π_α。
正文
Abstract:Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution $\pi_\alpha \propto \pi_{\text{base}}^\alpha$ via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to $\pi_\alpha$ under realistic compute budgets, and that their gains are largely recreated via simple Best-of-$N$ with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating $\pi_\alpha$, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match $\pi_\alpha$, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting $\pi_\alpha$, but in the simpler goal of extracting sequences the base model already deems likely.
| Subjects: | Machine Learning (stat.ML); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.13653 [stat.ML] |
| (or arXiv:2610.13653v1 [stat.ML] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13653 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Abiel Malkani Talwar [view email]
[v1]
Fri, 9 Oct 2026 14:29:28 UTC (235 KB)
来源：arXiv：cs.LG · arxiv.org