arXiv：cs.CL· Shrey Shah, Yinheng Li·· 10 小时前AI 评分46
Holdout Best-of-N：无偏评估及其代价
Holdout Best-of-N: Unbiased Evaluation and Its Cost
AI 导读
研究揭示复用选出 Best-of-N 赢家的分数会高估其期望奖励，并提出基于固定评分矩阵的 Holdout 估计器：仅当 J<K 时才能对期望评判奖励保持精确无偏。在独立高斯评分下，无偏 minimax 风险约为 σ²/√K，而允许偏差可将速率提升至 σ²/K；双候选情形下最小方差无偏估计的渐近常数为 1/(π√2)，Holdout 无需已知方差即可达到。
正文
Abstract:Reusing the scores that select a Best-of-$N$ winner can overstate its expected reward. We study evaluation from a fixed matrix of $K$ independent scores per candidate for a policy that selects using $J$ fresh scores. A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if $J<K$, for every pool size $M\ge N\ge2$. At $J=K-1$, the selector deepens as $K$ grows. For independent Gaussian scores with common variance and fixed $M\ge N\ge2$, the unbiased minimax risk in this regime is of order $\sigma^2/\sqrt K$, attained by Holdout; allowing bias improves the rate to $\sigma^2/K$. For two candidates, we derive the minimum-variance unbiased estimator at known variance and the sharp asymptotic unbiased minimax constant $1/(\pi\sqrt2)$, which Holdout attains without knowing the variance. The cyclic average over subsets and ties can be computed in $O(MK\log M)$ operations. At fixed selector depth, cyclic evaluation of bounded scores has $O(K^{-1})$ risk uniformly in pool size. The impossibility result concerns the fixed matrix: one additional fresh winner score permits unbiased evaluation of the all-$K$ policy.
| Comments: | 25 pages, 2 figures, 3 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.08719 [cs.CL] |
| (or arXiv:2610.08719v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08719 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yinheng Li [view email]
[v1]
Tue, 6 Oct 2026 17:26:05 UTC (665 KB)
来源：arXiv：cs.CL · arxiv.org