LLM 评审的暗电流：面向 LLM Judge 可靠性的分层零假设与效用检验

View PDF HTML (experimental)

Abstract:Passing an exact-clone null gate does not entail passing a matched target-selection utility gate. We call a non-tie preference between exact duplicates dark current. On 102 prespecified fresh RewardBench 2 rows, all four configuration--criterion cells report 0/102 clone preferences. Both neutral utility arms pass, but strict utility lower bounds of .539 and .745 miss the .75 working threshold. Conversely, re-evaluation of the same rows on two additional configurations yields a neutral gpt-oss utility pass with 2/102 clone preferences, leaving low-null behavior uncertified. A separately frozen 485-row extension passes both aggregate working gates under both criteria on one local raw-text configuration, yet strict Precise Instruction Following (Precise IF) alignment is 58/154 (.377). Thus matched null and utility results can diverge, and aggregate passage can coexist with weak subset alignment. Interpretation is conditional on the declared configurations, fixed source mixtures, source-label targets, and study-specific working models. The bounds are marginal, the subset rate is descriptive, and non-certification does not prove a defect.

Comments:	21 pages, 4 figures, 9 tables. Substantially revised: input-stratified null and target-selection utility audits, additional fixed-configuration studies, and an operational audit protocol. Supersedes v1, originally titled "LLM Judges Have Dark Current: A Psychometric Datasheet for LLM-as-a-Judge Evaluation."
Subjects:	Computation and Language (cs.CL); Instrumentation and Methods for Astrophysics (astro-ph.IM); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:	arXiv:2606.15610 [cs.CL]
	(or arXiv:2606.15610v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2606.15610 arXiv-issued DOI via DataCite

Submission history

From: Hiroyasu Usami [view email]
[v1] Sun, 14 Jun 2026 05:34:17 UTC (329 KB)
[v2] Mon, 12 Oct 2026 20:24:02 UTC (205 KB)