arXiv：cs.AI· Yuyang Dai, Yuting Tang, Futoshi Futami, Takashi Ishida·· 5 小时前
ProbabilityBench：用精确真实不确定性评估与校准模型
ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty
AI 导读
研究者推出 ProbabilityBench，在同行评审、隐蔽信息传递和信用风险三个模拟文本领域提供实例级精确类别概率，并给出将案例转写为自然语言备忘录仍严格保留生成概率的理论保证。
正文
Abstract:Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional class probabilities unknown. To address this, we introduce ProbabilityBench, a benchmark that provides exact, instance-level class probabilities across three simulated text-based domains: peer review, covert information transfer, and credit risk. By explicitly defining the full data-generating process for each environment, we analytically compute both exact class probabilities and Bayes error. Furthermore, we provide a theoretical guarantee that rendering these cases into natural-language memos strictly preserves their underlying generative probabilities. This exactness enables direct measurement of reported probability quality via KL divergence. Across 16 evaluated models, most models on the observed cost-quality Pareto frontiers are decision models, including Jev and Perplexity Decider. Finally, ProbabilityBench serves as a meta-evaluation framework, leveraging known ground-truth values to benchmark the accuracy of standard calibration metrics and Bayes-error estimators themselves.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.17016 [cs.LG] |
| (or arXiv:2610.17016v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17016 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Takashi Ishida [view email]
[v1]
Mon, 12 Oct 2026 15:58:15 UTC (126 KB)
来源：arXiv：cs.AI · arxiv.org