arXiv：cs.AI· Eunsu Kim, Haneul Yoo, Guijin Son, Hitesh Patel, Amit Agarwal, Alice Oh·· 23 小时前
BenchHub：面向组合感知的 LLM 评测框架，入选 NeurIPS 2026
Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub
AI 导读
BenchHub 是一个组合感知的 LLM 评测框架，将 benchmark 表示为实例级属性的分布，整合 54 个 benchmark、839K 样本、覆盖 10 种语言。它揭示动机相似的 benchmark 内部构成可能差异巨大、现有 benchmark 常含超出其宣称用途的可复用覆盖，且模型排名在不同重加权与重采样配置下会发生偏移。
正文
Abstract:LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.
| Comments: | Accepted to NeurIPS 2026 Evaluations & Datasets (E&D) Track |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2506.00482 [cs.LG] |
| (or arXiv:2506.00482v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2506.00482 arXiv-issued DOI via DataCite |
Submission history
From: Eunsu Kim [view email]
[v1]
Sat, 31 May 2025 09:24:32 UTC (963 KB)
[v2]
Mon, 12 Oct 2026 18:21:52 UTC (1,803 KB)
来源：arXiv：cs.AI · arxiv.org