arXiv：cs.LG· Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, Xinbo Gao·· 4 小时前
LoDA：面向 LoRA 持续学习的任务驱动子空间分解，兼顾知识共享与隔离
Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning
AI 导读
针对 LoRA 持续学习方法忽视任务共享方向、且旧任务零空间基在相关任务下近乎失效的问题，研究者从投影能量视角提出 Low-rank Decomposition and Adaptation（LoDA）。
正文
Abstract:Continual Learning (CL) requires models to sequentially adapt to new tasks without forgetting old knowledge. Recently, Low-Rank Adaptation (LoRA), a representative Parameter-Efficient Fine-Tuning (PEFT) method, has gained increasing attention in CL. Several LoRA-based CL methods reduce interference across tasks by separating their update spaces, typically building the new space from the estimated null space of past tasks. However, they (i) overlook task-shared directions, which suppresses knowledge transfer, and (ii) fail to capture truly effective task-specific directions since these ``null bases" of old tasks can remain nearly inactive for new task under correlated tasks. To address this, we study LoRA learning capability from a projection energy perspective, and propose Low-rank Decomposition and Adaptation (LoDA). It performs a task-driven decomposition to build general and truly task-specific LoRA subspaces by solving two energy-based objectives, decoupling directions for knowledge sharing and isolation. LoDA fixes LoRA down-projections on two subspaces and learns robust up-projections via a Gradient-Aligned Optimization (GAO) approach. After each task, before integrating the LoRA updates into the backbone, LoDA derives a closed-form recalibration for the general update, approximating a feature-level joint optimum along this task-shared direction. Experiments indicate that LoDA outperforms existing CL methods. Our code is available at this https URL.
| Comments: | Accepted by ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2603.00191 [cs.LG] |
| (or arXiv:2603.00191v5 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.00191 arXiv-issued DOI via DataCite |
Submission history
From: Lingfeng He [view email]
[v1]
Fri, 27 Feb 2026 02:31:00 UTC (7,144 KB)
[v2]
Sat, 2 May 2026 01:48:44 UTC (7,411 KB)
[v3]
Tue, 12 May 2026 07:35:28 UTC (7,411 KB)
[v4]
Sun, 24 May 2026 03:29:33 UTC (7,413 KB)
[v5]
Wed, 14 Oct 2026 08:43:34 UTC (7,413 KB)
来源：arXiv：cs.LG · arxiv.org