arXiv：cs.AI· Haiquan Hu, Yuzhu Liang, Weicheng Tang, Yanzeng Li, Yao Shi, Li Xu, Tian Wang·· 5 小时前
BSDProbe：通过条目级能力边界对 LLM 基准做结构性诊断
Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries
AI 导读
BSDProbe 提出了一个条目级基准结构诊断框架，通过沿经验排序的模型轴重复采样，构建边界位置、边界宽度、边界信号有效性和顺序一致性四维画像，在六个基准上识别出不同比例的可疑条目。以 Qwen3 作为筛选与参考轴，其归一化区分度比各基准最优基线平均相对提升 17.9%，与完整基准的排序一致性达到或超过所有基线。
正文
Abstract:Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their intended measurement purpose. BSDProbe provides an item-level framework for benchmark structural diagnosis that jointly characterizes correctness transitions and their supporting response evidence. Repeated responses along an empirically ordered model axis yield complementary correctness and normalized entropy trajectories, from which four-dimensional profiles of boundary position, boundary width, boundary-signal validity, and order consistency are constructed for benchmark-level aggregation. Experiments on six benchmarks identify varying proportions of potentially anomalous items. With Qwen3 as both the selection and reference axis, BSDProbe achieves a 17.9\% mean relative improvement in normalized discrimination ratio over the best baseline per benchmark and matches or exceeds all baselines in ranking agreement with the full benchmark.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.33196 [cs.AI] |
| (or arXiv:2609.33196v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.33196 arXiv-issued DOI via DataCite |
Submission history
From: Haiquan Hu [view email]
[v1]
Sun, 27 Sep 2026 04:30:18 UTC (481 KB)
[v2]
Mon, 12 Oct 2026 13:48:38 UTC (1,791 KB)
来源：arXiv：cs.AI · arxiv.org