arXiv：cs.LG· Fanyao Li, Yuxi Liu, Xiaohao Liu, He Chang, Zhaoxin Wang, Zhulin Tao·· 3 小时前
ReGRA：用残差梯度适配实现互补的低秩微调
ReGRA: Residual-Gradient Adaptation for Complementary Low-Rank Fine-Tuning
AI 导读
ReGRA 是一种两阶段低秩微调方法，在固定 rank 预算下先估计任务梯度并去除其在第一阶段更新子空间上的投影，再以残差梯度方向初始化第二阶段适配器，并用正交正则鼓励两次更新捕获互补方向。两次更新可合并进预训练权重，不增加推理开销；在多个基准上，ReGRA 优于同等总 rank 的方案，并可媲美 rank 翻倍的 LoRA 适配器。
正文
Abstract:Low-rank adaptation (LoRA) enables parameter-efficient fine-tuning by learning low-rank weight updates. Increasing rank expands adaptation capacity, but does not guarantee that all learned directions improve downstream performance. In a diagnostic experiment with rank-16 LoRA, we show that retaining only the top singular components achieves the best performance, whereas adding further components degrades it. Motivated by this, we propose Residual-Gradient Adaptation (ReGRA), a two-stage method for making more effective use of a fixed rank budget. After the first stage, ReGRA estimates task gradients on a calibration set and removes their projections onto the two-sided subspace induced by the first-stage update. It then aggregates the residual-gradient directions to initialize a second-stage adapter without changing the first-stage update. An orthogonality regularizer encourages the two updates to capture complementary directions. Both updates can be merged into the pretrained weights, incurring no additional inference overhead. Across diverse benchmark sets, ReGRA achieves stronger performance than alternatives with the same total rank and even matches LoRA adapters with twice the rank. Ablation studies and further analysis support the contribution of its core design choices. Code is available at this https URL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19700 [cs.LG] |
| (or arXiv:2610.19700v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19700 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Fanyao Li [view email]
[v1]
Wed, 14 Oct 2026 07:23:50 UTC (455 KB)
来源：arXiv：cs.LG · arxiv.org