arXiv：cs.LG· Sangyoon Bae, Sk Miraj Ahmed, Shinjae Yoo, Jiook Cha·· 3 小时前AI 评分44
预训练决定谱结构：基础模型 OOD 鲁棒性的架构与策略条件预测
Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models
AI 导读
研究发现基础模型的 OOD 鲁棒性编码在预训练权重的谱结构中，架构决定信息存储方式、预训练策略决定奖励目标，二者共同决定谱几何。仅凭预训练权重计算的统计量可作代理指标，其在同一（架构×策略）组合内方向稳定，样本内能以 92% 准确率排序模型对的 OOD 鲁棒性。按谱集中度行动可在 87.5% ID 保留率下将 OOD 差距缩小 24%。
正文
Abstract:Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separate robust models from fragile ones. We show the answer is encoded in the spectral structure of pretrained weights. Two forces shape that structure. Architecture determines how information is stored in weight matrices. Pretraining strategy determines what is rewarded. Together they set a spectral geometry that governs OOD robustness. We prove that the OOD accuracy gap is bounded by how tightly the source representations concentrate. A statistic computed from the pretrained weights alone serves as a proxy for that concentration. The direction of that proxy reverses between architecture families. We operationalize it: the direction is stable within one (architecture X strategy) combination, the finest grouping we test, which we call a cell. Pooled over 116 models spanning 7 modalities, a single statistic ranks OOD robustness weakly, because cells of opposite direction cancel. Within a cell, the statistic selected for it orders 92% of model pairs by OOD robustness in-sample. The selection does not leak the target: for each model family outside the matrix we logged the cell, metric and sign before running its OOD evaluation, and the predicted direction held in every case: EEG, genomic and protein. Acting on spectral concentration narrows the OOD gap by 24% at 87.5% ID retention. The diagnostic operates on released weights alone, so OOD robustness becomes checkable at model-selection time, before data or compute is committed to a target domain.
| Comments: | 10 pages, 3 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.09709 [cs.LG] |
| (or arXiv:2610.09709v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09709 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sangyoon Bae [view email]
[v1]
Wed, 7 Oct 2026 09:04:47 UTC (2,193 KB)
来源：arXiv：cs.LG · arxiv.org