arXiv：cs.AI· Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh, Jack Ji, Jithin Pradeep·· 6 小时前
LLM 评判器的评分格式决定了它能表达什么：一项关于指令微调数据筛选的实证研究
The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express
AI 导读
研究固定候选池、评判提示词、解码与选择预算，用 3 个 LLM 评判器以 5 种格式为 15,406 条 UltraFeedback 候选数据打分，微调 66 个模型并用 4 个评判器评估。
正文
Abstract:When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1--5 rating, 0--100 score, or comparison) is rarely justified or ablated. Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges. A 0--100 scale is worth only 7--12 equally used levels, two to three times a 1--5 scale rather than twenty, and a binary verdict leaves the top score tied in 77--78\% of prompts, so most selections are made at random. These properties are set by the format almost independently of the judge. Kendall's $\tau_b$ hides them: it ranks binary best while $\tau_a$ and top-1 agreement rank it last, because it discounts the ties a selector must break at random. Both comparative formats inherit a slot bias whose sign differs across judges. Downstream, curated data beat random selection under every evaluation judge, robust to length and markdown adjustment, but no ordering among formats is detectable. We propose four quantities that judge-based curation pipelines should report.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.17158 [cs.AI] |
| (or arXiv:2610.17158v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17158 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lovedeep Gondara [view email]
[v1]
Mon, 12 Oct 2026 17:36:39 UTC (57 KB)
来源：arXiv：cs.AI · arxiv.org