arXiv：cs.LG（机器学习，全量分类）· Sakin Kirti, Joel Zylberberg·· 14 小时前AI 评分33
信号-噪声分解（SNF）正则化将干扰变异隔离至可移除子空间
Signal-Noise Factorization Isolates Nuisance Variation into Removable Subspaces
AI 导读
研究者构建了强化信号-噪声分解（SNF）与信号-信号分解（SSF）的正则化器，在 CIFAR-100 分类任务上对比 L2 正则化基线，发现增强 SNF 提升模型性能而增强 SSF 无效。
正文
Abstract:Recent theoretical work identified fundamental properties of representation geometry that shape inference ability of deep neural networks. These include signal-noise factorization (SNF), the ability to segregate signal from noise, and signal-signal factorization (SSF), the ability to segregate task-specific and task-irrelevant signals. Here, we built regularizers that reinforce these two properties during training. We compared networks trained with these regularizers to $L_2$-regularized baseline networks on the CIFAR-100 classification task to understand how our regularizers shape representation geometry and impact performance on a well-known computer vision baseline. Enhancing SNF via regularization improved model performance but enhancing SSF did not. Motivated by biomedical applications, we investigated how our regularizers affected performance on the BloodMNIST dataset treated with MedMNIST-C corruptions at five severity levels, and found even larger performance gains using the SNF regularizer. To understand the mechanism by which SNF-regularization produces improved performance, we analyzed the nuisance subspaces across regularization regimes, finding that the SNF-regularized models represent noise in distinct subspaces, separate from class-relevant signal. Because this geometry is explicit, the dominant corruption-induced directions can be estimated on held-out data and projected out of the representations. This manipulation led to a substantial gain in accuracy. These results show that regularizers that enforce signal-noise factorization can produce substantial improvements on computer vision tasks that contain out-of-distribution image distortions at inference time. They also highlight how shaping representations affects model performance: isolating nuisance variables from categorical ones is more important than maintaining factorized representations of categorical variables.
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML) |
| ACM classes: | I.2.6 |
| Cite as: | arXiv:2610.00751 [cs.LG] |
| (or arXiv:2610.00751v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00751 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sakin Kirti [view email]
[v1]
Wed, 30 Sep 2026 21:46:40 UTC (1,946 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org