arXiv：cs.LG· Yu Sun, Junwei Zhou, Zuodong Xiang, Yike Zhang, Pengcheng Xu, Xinyu Guan, Ruoyun Ma, Hailu Xu·· 13 小时前
在 LoRA B 矩阵内，静态梯度归因在基于损失的遗忘上不如密度匹配的随机掩码
Static Gradient Attribution Underperforms a Density-Matched Random Mask on Loss-Based Forgetting Within LoRA B-Matrix
AI 导读
研究在 Qwen2.5-3B 上比较了静态梯度归因与随机掩码在 LoRA B 矩阵参数池中的连续学习表现，发现统一随机掩码在七个随机种子下一致取得更低的基于损失的反向迁移。作者指出此前比较存在参数池混淆：零初始化 LoRA 下对 A 的梯度恒为零，梯度掩码只选 B，而随机掩码覆盖 A 和 B。该优势集中在贡献 80% 聚合值的医疗任务上，且与稳定性—可塑性权衡一致。
正文
Abstract:Parameter-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data. We identify a parameter-pool confound in comparisons between initialization-time gradient masks and unrestricted random masks under standard zero-initialized LoRA ($B=0$): the gradient with respect to $A$ is identically zero ($\partial\mathcal{L}/\partial A=0$), so gradient masks select strictly from the $B$ matrix whereas unrestricted random masks draw from both $A$ and $B$. Such a comparison changes both the selection rule and the candidate parameter pool. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss-based backward transfer than gradient attribution across seven seeds on Qwen2.5-3B. The effect is concentrated on the medical task, which contributes 80\% of the aggregate; code and math move in the same direction, and legal shows no reliable difference. This advantage is consistent with a stability--plasticity trade-off under the tested task sequence, in which random masking slightly reduces single-task fitting. On the two discrete-option tasks (Medical and Legal), the mean accuracy-BWT difference is not statistically distinguishable from zero.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12744 [cs.LG] |
| (or arXiv:2610.12744v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12744 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yu Sun [view email]
[v1]
Thu, 8 Oct 2026 21:44:51 UTC (300 KB)
来源：arXiv：cs.LG · arxiv.org