arXiv：cs.LG· Hongyu Cao, Yanchi Liu, Kunpeng Liu, Xujiang Zhao, Wei Cheng, Zhengzhang Chen, Yanjie Fu, Haifeng Chen·· 4 小时前AI 评分35
GRADE：面向数据中心化小语言模型微调的梯度准入方法
Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning
AI 导读
针对 LoRA 微调小语言模型时梯度冲突、静态数据选择与子空间饱和三类问题，研究者提出数据中心化框架 GRADE，通过状态感知选择器持续准入与多任务梯度场对齐的样本，并用自校准步级门控拒绝接近饱和时的破坏性覆盖更新。在三种主干模型和七个异构指令数据集上，GRADE 在准确率与鲁棒性上均优于强数据选择和 PEFT 稳定化基线，且是唯一在每个架构上均稳定超过标准 LoRA 的方法。
正文
Abstract:LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data within a low-rank update subspace, making it vulnerable to three structural problems: conflicting gradients that cancel, static data selection that cannot track evolving learning dynamics, and subspace saturation that causes later updates to overwrite useful directions. We argue that effective adaptation therefore requires controlling which data-induced gradients enter the LoRA subspace and when. We propose GRADE (GRadient-Aligned Data-centric rEcipe), a data-centric framework combining two mechanisms: a state-aware selector that continually admits samples aligned with the evolving multi-task gradient field, and a self-calibrating step-level gate that rejects updates likely to cause destructive overwrite near saturation. Across three current-generation backbones and a heterogeneous seven-dataset instruction pool, GRADE outperforms strong data-selection and PEFT-stabilization baselines in accuracy and robustness. It is the only method to improve consistently over standard LoRA on every architecture, while producing more coherent gradient trajectories and less destructive overwrite. These results show that successful SLM adaptation depends not only on which data are selected, but also on which gradients are allowed to enter and persist in the constrained update subspace.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07553 [cs.LG] |
| (or arXiv:2610.07553v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07553 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hongyu Cao [view email]
[v1]
Tue, 6 Oct 2026 00:34:38 UTC (960 KB)
来源：arXiv：cs.LG · arxiv.org