arXiv：cs.LG（机器学习，全量分类）· Zailong Tian, Yanzhe Chen, Zhuoheng Han, Houfeng Wang, Lizi Liao·· 15 小时前AI 评分38
LoRA-Norm：面向 LoRA 的训练后归一化方法
Learn the Directions, Normalize the Gains: Post-Training Normalization for LoRA
AI 导读
研究者提出 LoRA-Norm，一种针对 LoRA 的训练后归一化方法，通过奇异值非线性变换的谱重平衡与核范数恢复，在保留已学方向的同时重新平衡其增益。该方法无需校准数据或额外训练，也不引入推理开销，在两个骨干模型和三个适配任务上同时提升了平均专业化与能力保留，优于所评估的训练后谱剪枝与梯度引导编辑配置。
正文
Abstract:While Low-Rank Adaptation (LoRA) enables efficient task specialization, its learned updates can compromise capabilities beyond the target task. We identify \textbf{adaptation imbalance}: a few singular directions dominate the trained update, leaving its performance sensitive to how gains are allocated. We argue that \textbf{learning where to adapt does not ensure that adaptation gains are well balanced}. This motivates \textbf{LoRA-Norm}, a post-training normalization method that retains learned directions while rebalancing their gains. LoRA-Norm combines spectral rebalancing, a fixed nonlinear transformation of singular values, with nuclear-norm restoration, which preserves the original total spectral mass. It requires no calibration data or additional training and introduces no inference overhead. Across two backbones and three adaptation tasks, LoRA-Norm improves average specialization and capability retention, outperforming the evaluated post-hoc spectral pruning and gradient-guided editing configurations on both measures. Stronger functional equalization brings no consistent additional gains, revealing that balancing adapter gains and equalizing their responses are distinct objectives.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02067 [cs.LG] |
| (or arXiv:2610.02067v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02067 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zailong Tian [view email]
[v1]
Thu, 1 Oct 2026 17:07:21 UTC (640 KB)
来源：arXiv：cs.LG（机器学习，全量分类） · arxiv.org