arXiv：cs.LG· Xiuyi Yang·· 4 小时前
多项式激活函数的损失曲面如何指导网络设计、初始化与训练
Trainable at Depth, Solvable at Small Size: What the Loss Surface of a Polynomial Activation Dictates for Designing, Initializing and Training Networks
AI 导读
研究提出多项式激活 f(z) = z + λz³（经截断并投影到单位球面）加一条未归一化的恒等路径，同一构造在两种规模下作答：作为深层堆叠，它在 MNIST 上无任何归一化机制即可训练到 100 层；作为小模型，其平方损失是可精确求解的多项式系统，小规模结果反过来规定了该堆叠的设计、初始化与训练方式。
正文
Abstract:What makes a deep network train, and what shape is its loss surface? We answer both with one construction at two scales: a polynomial activation f (z) = z + {\lambda}z 3 , clamped and projected onto the unit sphere, plus an unnormalized identity path. As a deep stack it trains to one hundred layers on MNIST with no normalization machinery; as a small model its square loss is a polynomial system we solve exactly. The small scale explains the large one: it prescribes how to design, initialize and train the stack.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.20069 [cs.LG] |
| (or arXiv:2610.20069v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20069 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiuyi Yang [view email]
[v1]
Wed, 14 Oct 2026 12:07:52 UTC (90 KB)
来源：arXiv：cs.LG · arxiv.org