arXiv：cs.AI· Yongxi Zhou, Lai Yun Choi, Jiaxi Wen, Wenbo Ye, Junwei Yao·· 5 小时前
LLM 代码评测中重复运行能买到多少？一项测得的方差上限
How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation
AI 导读
一项基于 16 个模型、99 道 LeetCode 风格题目（每题两个提示词、五次运行，共 15,840 次评判运行）的研究测出 LLM 代码评测的方差上限为 1+μ_w/σ_b²，模型间中位数 1.34（范围 1.05–1.71）。
正文
Abstract:A benchmark score is a measurement whose uncertainty has two sources: an aleatoric component from stochastic decoding and an epistemic component from the finite problem set. Using 16 models on 99 recent LeetCode-style problems with two prompts and five runs each (15,840 judged runs in a fully crossed design), we estimate both components and ask a design question: how much can repeated running buy? First, the pooled-binomial interval that repeated-run protocols invite answers neither question of interest---it is about $2\times$ too narrow for generalisation to new problems and $2\times$ too wide for the fixed problem set---and under correctly clustered intervals, 12 of 15 adjacent leaderboard pairs are statistically unresolved. Second, the variance ceiling $1+\mu_w/\sigma_b^2$, the largest factor by which repetition can reduce score variance, has median 1.34 across models (range 1.05--1.71), so repetition can narrow confidence intervals by at most 23% on this benchmark, and by 14% for the median model. Third, once the cost of adding verified problems is priced in, the precision-optimal repeat count is $R^{*}=\sqrt{\mu_w c_p/(\sigma_b^2 c_g)}$, and the variance curve is flat around it: on this benchmark the decisive reason to repeat is not precision but identification---two to three runs suffice to estimate the components that the ceiling and the fixed-benchmark interval require. All three are measurements of one benchmark at one decoding temperature, and we state explicitly what they do and do not license.
| Comments: | 17 pages, 4 figures, 11 tables. Substantially revised and retitled from v1; reframes the original repeated-run evaluation data as a variance decomposition, with expanded statistical validation and experimental details. Data and code: this https URL . Accepted at the NeurIPS 2026 Workshop on AI for Verifiable Coding (VeriCodeGen) |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Software Engineering (cs.SE) |
| Cite as: | arXiv:2606.00920 [cs.LG] |
| (or arXiv:2606.00920v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2606.00920 arXiv-issued DOI via DataCite |
Submission history
From: Yongxi Zhou [view email]
[v1]
Sat, 30 May 2026 23:03:05 UTC (755 KB)
[v2]
Sat, 10 Oct 2026 05:30:50 UTC (283 KB)
来源：arXiv：cs.AI · arxiv.org