arXiv：cs.CL· Mohammed Rawhani, Dervi\c{s} Karabo\u{g}a, \"Ozkan Ufuk Nalbanto\u{g}lu, Alper Ba\c{s}t\"urk, Bahriye Akay·· 3 小时前AI 评分34
MixedPEFT：用混合目标组合多种 PEFT 方法实现无监督域适应
MixedPEFT: Combining Multiple PEFT Methods with Mixed Objectives for Unsupervised Domain Adaptation
AI 导读
MixedPEFT 将可逆适配器与 LoRA 结合，在标注源域数据上优化分类、在无标注目标域数据上做掩码语言建模，仅更新 7% 的模型参数。在 MNLI 的 20 种域偏移上，该方法平均比参数高效 SOTA UDapter 高 1.41 个百分点，比全量微调的 DANN 高 1.26 个百分点，比 DSN 高 0.86 个百分点。
正文
Abstract:Applying pre-trained language models to new domains through full fine-tuning is computationally expensive and prone to catastrophic forgetting. To address this limitation, we introduce a novel parameter-efficient strategy for unsupervised domain adaptation that combines a custom PEFT architecture with mixed-objective training. The proposed method integrates invertible adapters with Low-Rank Adaptation (LoRA) and jointly optimizes classification on labeled source-domain data and masked language modeling on unlabeled target-domain data. This joint training scheme supports task adaptation while preserving knowledge of the target domain. We evaluate the method on the Multi-Genre Natural Language Inference (MNLI) dataset across 20 domain shifts. Our approach achieves average performance improvements of 1.41 percentage points over the parameter-efficient state-of-the-art UDapter, 1.26 percentage points over the fully tuned DANN baseline, and 0.86 percentage points over DSN, while updating only 7% of the model parameters. These findings establish a new state-of-the-art result for parameter-efficient unsupervised domain adaptation and demonstrate that carefully designed PEFT combinations with concurrent optimization can outperform both parameter-efficient and conventional fully tuned approaches.
| Comments: | 6 pages, 5 tables. Accepted at UBMK 2026. Builds upon our preliminary work presented at UBMK 2024. v2: revised text, references and tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.22272 [cs.CL] |
| (or arXiv:2606.22272v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.22272 arXiv-issued DOI via DataCite |
Submission history
From: Mohammed Rawhani M.Sc. [view email]
[v1]
Sat, 20 Jun 2026 23:45:19 UTC (115 KB)
[v2]
Wed, 7 Oct 2026 12:37:23 UTC (116 KB)
来源：arXiv：cs.CL · arxiv.org