arXiv：cs.LG· Junkang Liu·· 3 小时前AI 评分35
MuLoRA：面向持续学习的谱平衡低秩适配方法
MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning
AI 导读
针对 LoRA 在持续学习中出现的"谱可塑性坍缩"问题，研究者提出 MuLoRA，通过历史白化选取任务自适应基、并对动量更新做近似极正交化，联合控制容量分配与利用。在五个类增量基准、八种增量设置共 16 项指标中，MuLoRA 在其中 15 项取得最高平均准确率。
正文
Abstract:Low-rank adaptation (LoRA) provides a parameter-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity. We identify \emph{spectral plasticity collapse}: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low-rank space underutilized. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized. To address this problem, we propose \texttt{MuLoRA}, which jointly controls capacity allocation and utilization. First, historical whitening identifies input directions with strong current-task response relative to accumulated historical response, yielding a task-adaptive basis that remains fixed during training. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within theselected space. An orthonormal basis connects these mechanisms by transferring the factor-update spectrum exactly tothe induced weight update. We establish a max--min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross-step anisotropy. Across five class-incremental benchmarks and eight incremental settings, \texttt{MuLoRA} achieves the highest mean accuracy in 15 of 16 reported metrics.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02283 [cs.LG] |
| (or arXiv:2610.02283v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02283 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junkang Liu [view email]
[v1]
Thu, 1 Oct 2026 13:39:53 UTC (4,624 KB)
来源：arXiv：cs.LG · arxiv.org