arXiv：cs.LG（机器学习，全量分类）· Chen Cheng, Hilal Asi, John Duchi·· 15 小时前AI 评分35
标注者数量对模型校准有何影响？重新审视金标准标签
How many labelers do you have? A closer look at gold-standard labels
AI 导读
研究通过理论建模分析多标注者数据聚合流程，发现直接使用未聚合的原始标签信息，比使用清洗后的聚合标签更容易训练出良好校准的模型。使用聚合信息的估计器收敛稳健但速度较慢，而能有效利用全部标签的估计器在忠实于真实标注过程时收敛更快。理论预测在真实数据集上得到验证，该论文已被 JASA 接收。
正文
Abstract:The construction of most supervised learning datasets revolves around collecting multiple labels for each instance, then aggregating the labels to form a type of "true" label. We question the wisdom of this pipeline by developing a (stylized) theoretical model of this process and analyzing its statistical consequences, showing how access to non-aggregated label information can make training well-calibrated models more feasible than it is with cleaned labels. The entire story, however, is subtle, and the contrasts between aggregated and fuller label information depend on the particulars of the problem, where estimators that use aggregated information exhibit robust but slower rates of convergence, while estimators that can effectively leverage all labels converge more quickly if they have fidelity to (or can learn) the true labeling process. The theory makes several predictions for real-world datasets, including when non-aggregate labels should improve learning performance, which we test to corroborate the validity of our predictions.
| Comments: | 64 pages, 8 figures. Accepted to Journal of the American Statistical Association (JASA) for publication |
| Subjects: | Statistics Theory (math.ST); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2206.12041 [math.ST] |
| (or arXiv:2206.12041v3 [math.ST] for this version) | |
| https://doi.org/10.48550/arXiv.2206.12041 arXiv-issued DOI via DataCite |
Submission history
From: Chen Cheng [view email]
[v1]
Fri, 24 Jun 2022 02:33:50 UTC (867 KB)
[v2]
Tue, 4 Jun 2024 23:23:11 UTC (921 KB)
[v3]
Thu, 1 Oct 2026 12:40:28 UTC (895 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org