ScoringBench：用恰当评分规则评估表格基础模型的基准

View PDF HTML (experimental)

Abstract:Tabular foundation models such as TabPFN, TabICL, ExaOneTabular, and Nori already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics such as RMSE. This overlooks the distributional information these models are designed to provide. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules - including (weighted) CRPS, energy scores and interval scores - alongside standard point metrics. ScoringBench covers 101 regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demsar/autorank approach and a magnitude-preserving $z$-score ranking approach. Our evaluation of various model types, including (fine-tuned) tabular foundation models, gradient-boosted trees, and MLPs, shows that model rankings shift depending on the chosen scoring rule. Models that perform well on point-estimate metrics can rank lower on probabilistic ones. These findings demonstrate that evaluation metric selection is a core modeling decision and a domain-specific requirement for applications where certain types of errors carry higher consequences.

Subjects:	Artificial Intelligence (cs.AI)
Cite as:	arXiv:2603.29928 [cs.AI]
	(or arXiv:2603.29928v4 [cs.AI] for this version)
	https://doi.org/10.48550/arXiv.2603.29928 arXiv-issued DOI via DataCite

Submission history

From: Jonas Landsgesell [view email]
[v1] Tue, 31 Mar 2026 16:01:56 UTC (667 KB)
[v2] Mon, 27 Apr 2026 17:23:20 UTC (1,757 KB)
[v3] Mon, 4 May 2026 14:43:11 UTC (1,920 KB)
[v4] Mon, 12 Oct 2026 08:36:09 UTC (2,724 KB)