arXiv：cs.AI· Sida I. Wang·· 17 小时前
如何测量 LLM 评测中的所有噪声
Measuring all the noises of LLM Evals
AI 导读
研究定义并测量 LLM 评测中的三类噪声：给定问题上生成不同答案的预测噪声、采样问题的数据噪声，以及按总方差定律合并的总噪声，并提出对全部 LLM 配对做配对比较的 all-pairs comparison 方法，基于数百万条问题级预测进行测量。结果显示，多数评测在所有模型对上呈现高度可预测的总噪声水平，且配对预测噪声通常大于配对数据噪声，意味着通过平均降低预测噪声可显著提升统计功效。
正文
Abstract:Applying well-established statistical methods effectively to LLM evals requires consideration of model correlations and prediction noises. Leading evals often ignore data noise, have low statistical power, or make unnecessary trade-offs. We define and measure three types of noise: 1) prediction noise from generating different answers on a given question, 2) data noise from sampling questions, and 3) their combined total noise following the law of total variance. To emphasize relative comparisons and gain statistical power, we propose \emph{all-pairs comparison}, which applies paired comparison to all pairs of LLMs and measures all the noise components based on millions of question-level predictions across many evals and settings, revealing clear trends. First, most evals exhibit a characteristic and highly predictable total noise level across all model pairs. Second, paired prediction noise typically exceeds paired data noise, which means reducing prediction noise by averaging can significantly increase statistical power. By measuring all the noises together, we can assess eval results in context and make better empirical decisions.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (stat.ML) |
| Cite as: | arXiv:2512.21326 [cs.LG] |
| (or arXiv:2512.21326v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.21326 arXiv-issued DOI via DataCite |
Submission history
From: Sida I. Wang [view email]
[v1]
Wed, 24 Dec 2025 18:54:37 UTC (2,926 KB)
[v2]
Sun, 29 Mar 2026 00:57:41 UTC (2,938 KB)
[v3]
Tue, 13 Oct 2026 04:11:28 UTC (3,942 KB)
来源：arXiv：cs.AI · arxiv.org