arXiv：cs.AI· Yang Ba, Michelle V. Mancenido, Rong Pan·· 16 小时前
SCUR：LLM-as-a-Judge 的实例级可靠性评估方法
Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
AI 导读
研究者提出 Structured Cross-fitted Uncertainty Risk（SCUR），一种无需标签、基于评审面板行为估计 LLM-as-a-judge 实例级评估可靠性的方法。
正文
Abstract:LLM-as-a-judge evaluation commonly aggregates multiple judges' decisions into a majority vote, yet this panel decision process does not guarantee that the evaluation of an individual item is reliable. We propose Structured Cross-fitted Uncertainty Risk (SCUR), a label-free approach for estimating instance-level evaluation reliability based on the judge panel's behavior. A latent measurement model serves as the backbone for measurement system analysis, from which SCUR derives three complementary signals capturing panel uncertainty, measurement cross-entropy, and perturbation sensitivity. These three sources of evidence are combined into a risk score. We evaluate the SCUR approach on five real-world benchmarks using majority-vote error ranking, selective trust, and risk stratification. Furthermore, we show our approach to be more effective at detecting high-consensus risk items, which are the blind spots for many other LLM-as-a-judge evaluation methods.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.14029 [cs.LG] |
| (or arXiv:2610.14029v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14029 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yang Ba [view email]
[v1]
Wed, 30 Sep 2026 23:10:20 UTC (241 KB)
来源：arXiv：cs.AI · arxiv.org