arXiv：cs.LG· Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu·· 2 天前AI 评分49
语言模型能从测试时计算中获得多少收益？SELF-POT 基准给出量化答案
How Much Can Language Models Gain from Test-Time Computation?
AI 导读
研究者提出 SELF-POT 基准与评估框架，在竞赛数学、竞技编程和智能体工作流中衡量模型的测试时潜力，并按美元计费所有模型调用（含选择与评判）。在 350 个密封任务、五个低成本推理模型上，以 Claude Opus 5.5 Direct 为参照，收益取决于领域、选择规则与失败处理。
正文
Authors:Bangji Yang, Jingyuan Li, Jiajun Fan, Yi Evie Zhang, Ruihan Guo, Hongba Ma, Neil He, Chumeng Liang, Qinglong Zheng, Zhanghan Ni, Ge Liu
Abstract:How much can test-time computation improve a language model, and at what cost? Test-time scaling is widely proposed as a substitute for larger models, but existing comparisons mostly evaluate one domain at a time and rarely charge selection to the budget. We introduce SELF-POT, a benchmark and evaluation framework that measures the test-time potential of a model across competition mathematics, competitive programming, and agentic workflows. SELF-POT separates candidate coverage from final accuracy on static tasks, tracks correctness transitions under revision, and measures protocol completion alongside task success in agentic environments. Under a unified budget rule, it compares Direct inference with parallel sampling and self-revision under fixed multiples of the Direct budget, and charges every model call, including selection and critique, in dollars. This design supports two kinds of comparison: the gain a model obtains from additional inference, and a lower-cost model with additional inference against a stronger model. Across five low-cost reasoning models on 350 sealed tasks, with Claude Opus 5.5 Direct as the reference, the returns depend on the domain, the selection rule, and failure handling. When we replay the retained programming candidate pools, public-example selection raises correct submissions from 376 to 453 of 500 scheduled cells while saving 12-49% of logical API cost across models, and simply retaining an available candidate when judging fails recovers 61 submissions at unchanged cost. On identical mathematics pools, judging with fallback yields 186 correct submissions versus 182 for voting, while voting saves 12-21% of logical API cost. These controlled replays show how selection and failure handling change the gains realized from the same generated candidates, and they quantify the marginal value of a model judge.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.01110 [cs.LG] |
| (or arXiv:2610.01110v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.01110 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bangji Yang [view email]
[v1]
Thu, 1 Oct 2026 05:49:29 UTC (363 KB)
来源：arXiv：cs.LG · arxiv.org