arXiv：cs.CL· Tianjing Li, Wei Zhu·· 3 小时前
Diffu-LoRA：面向个性化扩散模型的新型低秩适配方法
Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models
AI 导读
Diffu-LoRA 是一种参数高效的低秩适配方法，通过门控机制为 Transformer 块的线性层学习逐层适配容量分配，并以双层优化和渐进剪枝满足指定的秩预算，同时保持预训练骨干冻结。在 Stable Diffusion 上基于 DreamBooth 等数据集的实验显示，其主体保真度与提示词对齐效果优于所评估的微调基线。
正文
Abstract:Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers. We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen. Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement. These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.10550 [cs.CL] |
| (or arXiv:2610.10550v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10550 arXiv-issued DOI via DataCite |
Submission history
From: Wei Zhu [view email]
[v1]
Thu, 24 Sep 2026 16:57:09 UTC (253 KB)
来源：arXiv：cs.CL · arxiv.org