arXiv：cs.AI· Zirui Peng, Heyi Zheng, Ziming Liu·· 16 小时前
重新思考激活函数：降低激活偏置可改善学习
Rethinking Activation Functions: Reducing Bias Can Improve Learning
AI 导读
一项 arXiv 研究提出"激活偏置"概念，指 ReLU、GELU、Swish、Sigmoid 等主流激活函数输出以正值为主、激活后均值非零。在受控模型上，激活偏置会加速初期损失下降，但伴随过冲、非平滑轨迹和更陡的损失曲面，降低偏置虽减慢初期学习却能得到更低的最终损失。在更大的视觉和语言模型上，较小的激活偏置同样与更好的后期性能相关。
正文
Abstract:Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.
| Comments: | 26 pages, 21 figures, 2 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14500 [cs.LG] |
| (or arXiv:2610.14500v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14500 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zirui Peng [view email]
[v1]
Sat, 10 Oct 2026 02:58:15 UTC (7,929 KB)
来源：arXiv：cs.AI · arxiv.org