arXiv：cs.LG· Paolo Mandica, Micha{\l} Brzozowski, Zuzanna Dubanowska, Neo Christopher Chung·· 2 天前AI 评分38
GPart：通过全局参数分区实现端到端等距微调
GPart: End-to-End Isometric Fine-Tuning via Global Parameter Partitioning
AI 导读
GPart 是一种高参数效率的微调方法，通过稀疏等距分区矩阵将 d 维可训练向量直接映射到完整权重空间，实现精确的端到端等距，仅需一个主超参数 d，检查点仅含可训练向量和随机种子。在自然语言理解、计算机视觉和数学推理基准上，GPart 在超低参数预算下匹配或超越现有 PEFT 方法。代码已开源。
正文
Abstract:Low-rank adaptation (LoRA) has become a dominant paradigm for parameter-efficient fine-tuning (PEFT) of large-scale deep learning models. However, its bilinear parameterization induces a parameter-dependent geometry: the mapping from trainable parameters to weight updates is not generally distance-preserving. Related methods that project a low-dimensional vector into LoRA's parameter space, such as Uni-LoRA, improve parameter efficiency, but the subsequent bilinear map breaks end-to-end isometry. We propose GPart (Global Partition fine-tuning), a highly parameter-efficient fine-tuning method that maps a $d$-dimensional trainable vector directly into the full weight space through a sparse, isometric partition matrix. GPart retains a fixed global parameter-sharing prior while removing the additional low-rank reconstruction used by LoRA-based methods. This yields a simple parameterization with a single main hyperparameter ($d$), exact end-to-end isometry, and a minimal checkpoint representation consisting of the trainable vector and a random seed. GPart builds on the premise of effective fine-tuning within random low-dimensional subspaces of the full weight space without requiring a low-rank matrix factorization. Across natural language understanding, computer vision, and mathematical reasoning benchmarks, GPart matches or improves over existing PEFT methods at ultra-low parameter budgets. Beyond offering mathematical tractability and memory efficiency, the direct linear parameterization of GPart streamlines model selection and paves the way for compact adapter composition. Overall, GPart provides an elegant and competitive alternative for fine-tuning under small parameter budgets, with a fixed and predictable geometry between trainable coordinates and weight-space updates.
| Comments: | Code available at this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.14841 [cs.LG] |
| (or arXiv:2605.14841v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.14841 arXiv-issued DOI via DataCite |
Submission history
From: Paolo Mandica [view email]
[v1]
Thu, 14 May 2026 13:46:04 UTC (6,019 KB)
[v2]
Thu, 1 Oct 2026 11:55:46 UTC (6,055 KB)
来源：arXiv：cs.LG · arxiv.org