arXiv：cs.CL· Sangdo Han, Moonsoo Kim·· 11 小时前
无标签评分标准澄清让 LLM 评委更一致，但未更贴近人类判断
Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans
AI 导读
一项无标签方法将 LLM 评委间的分歧转化为评分标准修订：把每条标准拆成子标准，按跨评委分歧排序，由编辑模型只重写争议最大的条款，全程不使用人类标签。
正文
Abstract:LLM judges are increasingly deployed as panels, and disagreement among panel members is often read as a symptom of an unclear rubric. We test a label-free procedure that turns this disagreement into rubric repairs. The procedure decomposes each rubric into sub-criteria, ranks them by cross-judge disagreement on a development split, and has an editor model rewrite only the most contested ones. An additional stage that moves text-checkable sub-criteria to deterministic checkers transferred none in our evaluation. Human labels are never used to select or edit clauses; they are used only to evaluate the outcome. We pre-specified four hypotheses in a version-controlled protocol and evaluated six rubric conditions with a panel of five judges from three providers on SummEval, the HealthBench meta-evaluation, and USR Topical-Chat, for a total of 129,770 judge requests, excluding retries. On SummEval, the procedure raised inter-judge Krippendorff's $\alpha$ by 0.069 (95% CI 0.046 to 0.092) over the original rubric and by 0.074 over a control that applied the same edits to an equal number of randomly chosen clauses that were not selected. Agreement with expert ratings showed no detectable change (Kendall $\tau$ difference -0.007, 95% CI -0.037 to 0.021). On HealthBench and USR, neither inter-judge agreement nor human agreement met the preregistered effect criteria. In a preregistered secondary comparison, human agreement after editing the contested clauses was lower than after editing the control clauses on all three datasets in point estimate, although no interval excluded zero. Selecting clauses by disagreement makes judges agree with each other more than editing arbitrary clauses does, but in our setting the added consensus did not bring the judges closer to human annotators. We will release code, frozen rubrics, item identifiers, and judge scores.
| Comments: | 12 pages, 12 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.13303 [cs.CL] |
| (or arXiv:2610.13303v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13303 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sangdo Han [view email]
[v1]
Fri, 9 Oct 2026 09:49:31 UTC (48 KB)
来源：arXiv：cs.CL · arxiv.org