模型发布门禁该有多严？测试相关性如何推高可靠性成本

View PDF HTML (experimental)

Abstract:Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many models that would have served users well, and it does not say how trustworthy a passing model actually is. We treat the release gate as a design problem: choose how many tests a model must pass so that cleared models meet a stated reliability target, while keeping as many good models as possible. A two-class latent-factor model makes both costs explicit and reduces each calculation to a one-dimensional integral. We prove that when both classes share the same latent correlation, a stricter gate always raises reliability, so the gate that keeps the most good models is the most lenient one that still meets the target. Under pass-all gating, any reliability target short of perfection is attainable within the model, but the share of good models kept tends to zero as the suite grows. Correlation between tests sets the price. In one configuration, a 99 percent target needs 8 independent tests, but 74 tests at a latent correlation of 0.3 and 5,182 at 0.5, where the gate keeps fewer than one good model in ten. We also give a validation procedure, built on exact binomial bounds, that certifies a gate from labelled data even when the gate is chosen from a fixed shortlist.

Comments:	6 pages, 1 figure, 4 tables. Submitted to ACDSA 2027
Subjects:	Machine Learning (stat.ML); Machine Learning (cs.LG); Methodology (stat.ME)
MSC classes:	62P30, 62F25, 62H25
ACM classes:	I.2.6; G.3; D.2.5
Cite as:	arXiv:2610.00993 [stat.ML]
	(or arXiv:2610.00993v1 [stat.ML] for this version)
	https://doi.org/10.48550/arXiv.2610.00993 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Marco Pollanen [view email]
[v1] Thu, 1 Oct 2026 03:27:54 UTC (98 KB)