arXiv：cs.AI· Qi Cao, Kangning Liu, Xuan Kan, Shunwen Tan, Yang Pei, Dake Chen, Yatai Ji, Zixuan Ye, Yuanpeng Tu, Daniel Li, Junbiao Tang, Pengtao Xie, Zihao He·· 6 小时前AI 评分45
JudgeProfile：理解并引导 LLM 评判者的主观性
JudgeProfile: Understanding and Steering Subjectivity in LLM Judges
AI 导读
研究团队提出 JudgeProfile 框架，将 LLM 评判拆解为感知（跨清晰度、正确性等属性比较两个回答）与优先级（各属性对最终选择的影响权重）两部分，并构建了包含 50,013 组回答对、由 21 个 LLM 评判者在 87 个属性上评估的 SubjectiveSet 数据集。
正文
Authors:Qi Cao, Kangning Liu, Xuan Kan, Shunwen Tan, Yang Pei, Dake Chen, Yatai Ji, Zixuan Ye, Yuanpeng Tu, Daniel Li, Junbiao Tang, Pengtao Xie, Zihao He
Abstract:LLM judges are inherently subjective, often favoring different responses in pairwise comparison when neither option is objectively wrong. To study this subjectivity, we introduce JudgeProfile, a framework that dissects LLM evaluation into perception (how a judge compares two responses across specific attributes like clarity, correctness, and detail) and prioritization (how much each attribute influences the final choice). We curate SubjectiveSet, a dataset of 50,013 response pairs from 17 public data sources, evaluated by 21 LLM judges across 87 attributes. We find a hidden consensus in perception: judges frequently agree on attribute judgments even when their overall choices diverge. Building on this separation, we first characterize each judge's prioritization using attribute weights estimated from its own overall choices. These weights differ across judges even when estimated from the same attribute judgments. We then learn new weights from reference labels to adapt their decisions to a target evaluation standard. Reweighting perceived attributes improves average held-out agreement with reference labels from 66.48% to 71.97%, outperforming fine-tuning and rubric prompting. Our findings show that understanding and steering the subjectivity of LLM judges requires attention not only to what they perceive, but also to how they prioritize it.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.36705 [cs.AI] |
| (or arXiv:2609.36705v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.36705 arXiv-issued DOI via DataCite |
Submission history
From: Qi Cao [view email]
[v1]
Tue, 29 Sep 2026 04:43:28 UTC (741 KB)
[v2]
Tue, 6 Oct 2026 00:21:07 UTC (741 KB)
来源：arXiv：cs.AI · arxiv.org