arXiv：cs.LG· Maoqi Liu, Quan Fang, Yufei He·· 7 小时前AI 评分42
CoDe-LoRA：通过知识巩固与解耦缓解 LLM 持续学习中的正交性困境
CoDe-LoRA: Mitigating the Orthogonality Dilemma in Continual Learning of LLMs via Knowledge Consolidation and Decoupling
AI 导读
针对 O-LoRA 等正交投影方法存在的“正交性困境”，研究者提出无回放的持续学习方法 CoDe-LoRA，将学习过程拆分为通用知识巩固与任务特定知识解耦，并用自适应零空间投影与语义路由平衡知识积累和任务适配。在四个骨干模型和三个持续学习基准上，CoDe-LoRA 取得最佳平均准确率，该工作已被 EMNLP 2026 主会接收，代码已开源。
正文
Abstract:Continual learning (CL) is essential for Large Language Models (LLMs) to sequentially adapt to evolving tasks. To mitigate catastrophic forgetting, recent advances implement low-rank adaptation with orthogonal projections (e.g., O-LoRA) to isolate task parameters. However, we reveal that such strict geometric constraints trigger an "Orthogonality Dilemma": rigid parameter isolation impedes the transfer and accumulation of shared representations across semantically related tasks. In this work, we propose a new replay-free method, called Consolidation and Decoupling LoRA (CoDe-LoRA), for CL of LLMs. CoDe-LoRA disentangles the learning process into Consolidating Universal Knowledge and Decoupling Task-Specific Knowledge. To achieve this, CoDe-LoRA leverages an adaptive null space projection mechanism and semantic routing to balance knowledge accumulation with task-specific adaptation. Experimental results across four backbones and three CL benchmarks show that CoDe-LoRA achieves the best average accuracy. Our code is available at this https URL.
| Comments: | Accepted to EMNLP 2026 (Main Conference) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.08312 [cs.AI] |
| (or arXiv:2610.08312v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08312 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Maoqi Liu [view email]
[v1]
Tue, 6 Oct 2026 13:17:25 UTC (5,678 KB)
来源：arXiv：cs.LG · arxiv.org