arXiv：cs.LG· Mijung Park·· 13 小时前
OMA：面向异构模型输出的可优化模型平均集成方法
Optimizable Model Averaging for Ensembling Heterogeneous Model Outputs
AI 导读
研究者提出 Optimizable Model Averaging（OMA），一种黑盒集成方法，为每个候选模型学习一个非负标量权重，在冻结全部模型参数的前提下对预测概率加权平均，从而支持架构不同的模型集成。该方法通过最小化带正则化的预测熵来学习权重，并考察正则化对校准的影响，在部分配置下观察到更低的预期校准误差，性能与校准的权衡优于模型输出平均。
正文
Abstract:Foundation model ensembles can improve accuracy and robustness, but full fine-tuning and parameter-space merging are often expensive or impossible when candidate models have different architectures. We propose Optimizable Model Averaging (OMA), a black-box approach that learns nonnegative model-combination weights, one scalar coefficient per model, to average the candidate models' predictive probabilities, leaving all model parameters frozen, by minimizing regularized predictive entropy. A central challenge is that minimizing predictive entropy can favour confidently incorrect models: increased confidence need not imply improved accuracy or calibration. We therefore examine regularization as a mechanism for controlling adaptation and empirically evaluate its effect on calibration, observing lower expected calibration error in selected configurations, alongside better performance-calibration trade-offs than model output averaging. As foundation models continue to advance, these approaches will enable the incorporation of future, potentially much better models to improve performance on challenging text and image classification tasks.
| Subjects: | Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2505.21857 [cs.LG] |
| (or arXiv:2505.21857v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.21857 arXiv-issued DOI via DataCite |
Submission history
From: Mijung Park [view email]
[v1]
Wed, 28 May 2025 01:03:28 UTC (270 KB)
[v2]
Thu, 8 Oct 2026 21:34:33 UTC (2,255 KB)
来源：arXiv：cs.LG · arxiv.org