arXiv：cs.LG（机器学习，全量分类）· Gongyue Zhang, Honghai Liu·· 14 小时前AI 评分34
逻辑与记忆的冲突：浅层 MLP 中的高阶交互学习
The Conflict Between Logic and Memory: Learning Higher-Order Interactions in Shallow MLPs
AI 导读
研究在单隐层 MLP 上用合成任务考察高阶交互学习，发现网络能拟合训练样本却学不到生成标签的规则。在 order-2–4 扫描中，SGD、Adam、Muon 在二阶均达 100% 峰值测试准确率，三阶分别为 96.25%、50.87%、76.82%，Muon 在四阶达 99.21%。冻结连接独立干扰输入的第一层权重可将 AdamW 第 10 轮准确率从 44.73% 提升至 95.07%。
正文
Abstract:A network can fit its training examples while failing to recover the rule that generated their labels. We examine this separation in single-hidden-layer multilayer perceptrons (MLPs), using synthetic tasks that control interaction order and the presence of nuisance inputs. We establish elementary benchmark properties: pure parity contains no predictive lower-order marginals, admits an exact Bayes posterior, and can be represented on clean latent inputs by a width-$k$ ReLU network. Experiments then identify distinct optimization outcomes. In a matched order-2--4 sweep, SGD, Adam, and Muon all reach 100\% peak test accuracy at order two; at order three they reach 96.25\%, 50.87\%, and 76.82\%, respectively, while Muon reaches 99.21\% at order four. In a separate mixed-order task, freezing only the first-layer weights connected to independent nuisance inputs raises AdamW's epoch-10 accuracy from 44.73\% to 95.07\%. Removing the same inputs only at test time raises it to 48.38\%. Thus, nuisance-weight learning changes the training outcome beyond its immediate effect on prediction. Bias interventions expose a connection between target symmetry and shallow ReLU representations. In a compact signal-only regime, both SGD and Muon learn orders five through eight, with higher SGD peak accuracy at orders nine through eleven. Together, the results show how optimization and nuisance learning constrain the higher-order rules realized by a shallow network.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.00403 [cs.LG] |
| (or arXiv:2610.00403v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00403 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gongyue Zhang [view email]
[v1]
Wed, 30 Sep 2026 13:08:59 UTC (301 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org