arXiv：cs.CL· Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li, Yuan Lu, Zhenhao Chen, Jiajun Fan, Shuai Shao, Chunyuan Zheng, Hao Wang·· 16 小时前
并非所有标准都同等重要：面向 Rubric 式 LLM 评测的重要性感知标准加权（ICW）
Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
AI 导读
针对 Rubric 式 LLM 评测中现有方法对所有标准统一加权、使次要且重叠的标准与关键标准同权而产生误导性评分的问题，研究者提出重要性感知标准加权（ICW）。该方法将 LLM 对所有标准的整体判断拆解为一组两两重要性比较，检查并修正不一致的比较，再从比较矩阵的主特征向量导出权重。实验显示，ICW 持续优于统一加权和竞争性加权方法，并在下游强化学习中转化为更好的策略模型。
正文
Abstract:Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19915 [cs.CL] |
| (or arXiv:2610.19915v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19915 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Haocheng Yang [view email]
[v1]
Wed, 14 Oct 2026 10:05:43 UTC (506 KB)
来源：arXiv：cs.CL · arxiv.org