arXiv：cs.LG· Zhanhong Jiang, Soumik Sarkar·· 13 小时前
pFedDPO：用 LoRA 适配器实现个性化联邦直接偏好优化
Personalized Federated Direct Preference Optimization via LoRA Adapters
AI 导读
针对 FedDPO 假设所有客户端收敛到同一策略、与现实中异构偏好分布相矛盾的问题，研究者提出 pFedDPO，让每个客户端在共享全局基座模型上保留一个私有 LoRA 适配器，基座模型每轮通过 FedAvg 同步，适配器留在本地捕捉客户端特有偏好。
正文
Abstract:Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity $\kappa^2$ in the FedDPO bound with a residual heterogeneity $\kappa^2_{\mathrm{eff}}$, which is no larger than $\kappa^2$ when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.
| Comments: | 15 pages, 11 figures, AAAI Fall Symposium Series 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12885 [cs.LG] |
| (or arXiv:2610.12885v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12885 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhanhong Jiang [view email]
[v1]
Fri, 9 Oct 2026 02:20:29 UTC (276 KB)
来源：arXiv：cs.LG · arxiv.org