arXiv：cs.AI· Daniel T. Bernstein, Chase Goddard, Luca Di Carlo, Vudtiwat Ngampruetikorn, David J. Schwab·· 19 小时前
Fisher 信息与神经网络记忆的几何结构
Fisher information and the geometry of memorization in neural networks
AI 导读
研究在 CIFAR-10 上训练并加入可控噪声的视觉模型中发现，将模型权重投影到 Fisher 特征向量的主方向上，可将通用任务性能与噪声样本记忆解耦，且这种解耦在不同层间表现不同。
正文
Abstract:A fruitful approach towards understanding generalization in machine learning involves characterizing how models separate generalizable structure from idiosyncratic or even corrupted features of training data. Fisher information has emerged as a diagnostic tool for this separation, leveraging loss geometry to distinguish generalization from memorization. Here we investigate this connection in vision models trained on CIFAR-10 with controlled noise that can be learned only through memorization. We show that projecting model weights onto the top Fisher eigenvectors decouples general task performance from noisy sample memorization, and we investigate how this decoupling presents differentially across layers. We show that the same phenomena are present in small transformers trained to solve a synthetic in-context learning task, and are reproduced in closed form in a simple, analytically solvable linear model. Our results reveal that Fisher information captures aspects of the geometric structure underlying how neural networks allocate capacity between generalization and memorization.
| Comments: | 20 pages (10 pages main text), 11 figures (6 figures main text), NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Disordered Systems and Neural Networks (cond-mat.dis-nn); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.16161 [cs.LG] |
| (or arXiv:2610.16161v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16161 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Daniel Bernstein [view email]
[v1]
Mon, 12 Oct 2026 04:34:54 UTC (2,177 KB)
来源：arXiv：cs.AI · arxiv.org