HuggingFace Daily Papers·· 1 天前
权重归一化训练中的有效学习率调度：AIR 调度方法研究
Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training
AI 导读
研究指出权重归一化（WN）训练的预训练损失收益主要来自诱导的有效学习率（ELR）轨迹而非归一化本身，匹配 ELR 轨迹后 WN 与权重衰减（WD）的训练损失动态基本一致。
正文 · 原文
Abstract:Weight-normalized (WN) training is emerging as a compelling alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. We show that these gains are driven primarily by the induced effective learning rate (ELR) trajectory rather than by normalization alone. Matching ELR trajectories largely collapses the training loss dynamics of WN and WD. Schedule design should therefore target ELR rather than the nominal learning rate (LR) alone. Building on this principle, we propose AIR (Affine Inverse-square-Root decay), which combines a theoretically derived $1/\sqrt{t}$ decay with an affine adjustment to satisfy prescribed peak and terminal LRs over a finite training horizon. We further identify late-stage ELR scheduling as a key source of its gains, because a shared late schedule attenuates the effect of earlier LR choices. Experiments on 128 B-token Mixture-of-Experts pretraining demonstrate that AIR outperforms cosine and other candidate schedules in training loss, downstream performance, and hyperparameter robustness, reducing terminal training loss by approximately 0.01 relative to cosine at matched cumulative LR. Applying the same ELR trajectory to WD reproduces the corresponding training dynamics, showing that the benefits extend beyond fixed-norm training and establishing ELR as the principled scheduling object.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.40362 [cs.LG] |
| (or arXiv:2610.40362v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40362 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qifan Yu [view email]
[v1]
Thu, 29 Oct 2026 16:15:08 UTC (1,547 KB)
来源：HuggingFace Daily Papers · arxiv.org