arXiv 论文提出 RIFT 分类法量化评估 rubric 质量并识别九种失败模式

View PDF HTML (experimental)

Abstract:Rubrics distill notions of expert quality and measure agent performance. However, the quality of rubrics themselves have not been systematically measured and are often left to downstream performance. We import apparatuses from measurement theory built for exactly this: quantitative signals based on the rubric's content, and introduce the RubrIc-Failure Taxonomy (RIFT), of nine possible ways a rubric fails, organized under reliability and content validity. Every mode leaves a distinct signature. To show the signals track failure causally, we seed 720 corruptions, injecting each RIFT mode into clean rubrics at known severity levels. A linear probe over the signals identifies which mode was injected at $75.0\%$ accuracy, beating $56.7\%$ for a frontier model asked to name the failure directly. Surprisingly across GDPval and Terminal-Bench, 10 of 48 expert-authored rubrics weight their criteria backwards, putting more of the score on requirements an expert panel judged less essential. This means a response can fail what matters most and still be graded well. This paper serves as a comprehensive guide on how to understand failure modes in rubrics and create better versions using quality signals, causal experiments, and provides a taxonomy with its rules and examples.

Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2604.01375 [cs.AI]
	(or arXiv:2604.01375v4 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2604.01375 arXiv-issued DOI via DataCite

Submission history

From: Charles Dickens [view email]
[v1] Wed, 1 Apr 2026 20:34:43 UTC (53 KB)
[v2] Mon, 20 Apr 2026 23:25:41 UTC (49 KB)
[v3] Thu, 1 Oct 2026 16:45:07 UTC (1,266 KB)
[v4] Fri, 2 Oct 2026 17:40:11 UTC (1,266 KB)