arXiv：cs.AI· Kakei Yamamoto, So Kuroki·· 16 小时前
PRISM：用反事实探针校准黑盒 LLM 评审
PRISM: Calibrating Black-Box Judges with Counterfactual Probes
AI 导读
针对 LLM 评审打分与人类判断系统性不一致的问题，研究者提出 PRISM（Probe-based Residual Inference of Scoring Misalignment），通过固定被评内容、测量属性被强调或弱化时的分数变化，并用岭回归在人类标签上学习对基线分数的校正。
正文
Abstract:Large language models are increasingly used as judges to reduce reliance on costly human evaluation, yet their scores can systematically disagree with human judgments. We ask whether responses to an explicitly imposed evaluative bias can help correct a judge's implicit scoring biases. We introduce PRISM (Probe-based Residual Inference of Scoring Misalignment), which holds the evaluated content fixed and measures score changes when attributes are emphasized or de-emphasized. A local sensitivity model motivates representing these responses as a surrogate Jacobian. Ridge regression on human labels learns a correction to the baseline score. The resulting scoring protocol is rubric-free: it uses overall scores without requesting criterion-level ratings. In the main Jev and Sol comparisons on SummEval summarization and HANNA story evaluation, PRISM and its learned-baseline variant significantly improve agreement with human rankings over uncalibrated scores while remaining competitive with rubric-based methods. A controlled SummEval comparison yields similar agreement for rubric-derived quality probes and style probes, supporting calibration without reproducing rubric criteria. The learned coefficients reveal shared and opposing correction directions across judges.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.18389 [cs.LG] |
| (or arXiv:2610.18389v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18389 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kakei Yamamoto [view email]
[v1]
Tue, 13 Oct 2026 13:39:50 UTC (351 KB)
来源：arXiv：cs.AI · arxiv.org