arXiv：cs.CL· Feifei Qian, Xuan Chen, Lixin Cui, Chaoyang Mei, Fanlin Meng, Shuyao Xiao, Ming Cui, Junxi Yin, Lu Bai·· 11 小时前
从结构到效度：机制引导的 LLM 评分标准生成与精炼
From Structure to Validity: Mechanism-Guided Generation and Refinement of LLM Rubrics
AI 导读
研究对 LLM 评分标准设计做了机制分析，发现结构化组织能构建更全面的评分条目，仅靠语义规范不足以保证条目级判断有效，复杂分数聚合带来的额外收益有限。基于此提出 ARTS-Hybrid，结合自适应树构建与语义优先、响应校准的精炼方法，在多个基准上同时提升了端到端评测表现和评分标准本身的质量，其生成的评分标准还可为模型优化提供有效信号。
正文
Abstract:Rubrics have been widely used to evaluate and optimize language models on open-ended tasks. However, existing work primarily focuses on generating better rubrics, with limited understanding of what makes rubric design effective. In this work, we conduct a systematic mechanistic analysis of rubric design across criterion organization, criterion operationalization, and score aggregation. Our analysis shows that structural organization enables more comprehensive criterion construction, semantic specification alone does not ensure effective criterion-level judgments, and sophisticated score aggregation provides limited additional gains. Guided by these findings, we introduce Adaptive Rubric Tree Search with Hybrid refinement (ARTS-Hybrid), which combines adaptive tree construction with semantic-first, response-calibrated refinement. Experiments across benchmarks show that ARTS-Hybrid improves both end-to-end evaluation performance and intrinsic rubric quality. Further experiments suggest that, beyond evaluation, rubrics generated by ARTS-Hybrid provide effective signals for model optimization.
| Comments: | Submitted to ICLR 2027 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13375 [cs.CL] |
| (or arXiv:2610.13375v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13375 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xuan Chen [view email]
[v1]
Fri, 9 Oct 2026 10:48:51 UTC (605 KB)
来源：arXiv：cs.CL · arxiv.org