arXiv：cs.AI· Yunji Wang, Junjie Yao, Linyu Liu, Pinyan Lu, Zhi-Qin John Xu·· 4 小时前
概率签名动力学：两层网络中模加运算学习机制的解析
Probability-Signature Dynamics: Unpacking Modular Addition Learning Within Two-Layer Networks
AI 导读
研究提出用"概率签名"解释两层网络在模加任务中为何会自发形成傅里叶结构化表示，这些签名是循环移位算子，经离散傅里叶变换对角化后得到近似解耦的傅里叶模态动力学，从而说明傅里叶稀疏性、频率匹配与相位对齐的出现。该框架还解释了标签噪声下受损样本早期损失下降快于干净样本的现象，即噪声增加条件标签碰撞、强化早期共享坐标。该方法可推广至其他算子，在 XOR 实验中观测到预测频率。
正文
Abstract:Neural networks trained on modular addition tasks often develop Fourier-structured representations that support exact generalization. While prior work has identified these Fourier circuits, the mechanism by which gradient-based training selects them from the data distribution remains unclear. We address this question using probability signatures, which express leading gradient interactions through conditional statistics of the training distribution. For modular addition, these signatures are cyclic shift operators and are diagonalized by the discrete Fourier transform, yielding approximately decoupled Fourier-mode dynamics. This explains the emergence of Fourier sparsity, frequency matching, and phase alignment. The same framework resolves a puzzle under label noise: corrupted examples can show faster early loss decrease than clean examples, despite lacking a coherent generalization rule. We show that noise increases conditional label collisions, strengthening early shared-coordinate reinforcement. Finally, this method can be applied to other operators. Taking XOR as an example, we observed the predicted frequency in experiments.
| Comments: | 36 pages, 18 figures. Submitted to ICLR 2027 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.11833 [cs.AI] |
| (or arXiv:2610.11833v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.11833 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yunji Wang [view email]
[v1]
Thu, 8 Oct 2026 12:27:35 UTC (760 KB)
来源：arXiv：cs.AI · arxiv.org