arXiv：cs.LG· Zhilin He, Gauri Joshi·· 7 小时前AI 评分36
FedGuide：异构联邦强化学习的扩散先验对齐与价值基线引导
FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning
AI 导读
FedGuide 是一个面向异构联邦强化学习的框架，用扩散先验作为行为模型为异构本地策略学习提供个性化数据分布支持，并通过 OT-MoE 聚合扩散先验而非直接平均本地策略。
正文
Abstract:Federated Reinforcement Learning (FRL) enables collaborative policy learning across distributed agents with heterogeneous environments. While recent methods based on variance reduction, divergence penalization, and momentum optimization improve FRL under heterogeneous settings, they still primarily synchronize policy or value-network parameters and do not explicitly address distributional mismatch among heterogeneous clients. Therefore, we propose \textbf{FedGuide}, a FRL framework that uses diffusion priors as behavior models to provide personalized data supported distributions for heterogeneous local policy learning. Instead of directly averaging local policies, FedGuide aggregates those diffusion priors through Optimal-Transport Mixture-of-Experts (OT-MoE), preserving heterogeneous behavior modes in distribution space. It further develops a Distribution Correction Estimation (DICE) value baseline to provide low-variance, return-aware guidance for local policy improvement. Experiments across heterogeneous environments show that FedGuide outperforms representative FRL methods in client-average returns, final-round performance, and worst-round robustness, while maintaining stable learning under stronger heterogeneity.
| Comments: | Accepted to the Conference on Robot Learning (CoRL), 2026. Spotlight presentation |
| Subjects: | Machine Learning (cs.LG); Robotics (cs.RO) |
| Cite as: | arXiv:2609.18964 [cs.LG] |
| (or arXiv:2609.18964v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.18964 arXiv-issued DOI via DataCite |
Submission history
From: Zhilin He [view email]
[v1]
Wed, 22 Jul 2026 05:27:53 UTC (8,734 KB)
[v2]
Tue, 6 Oct 2026 04:21:16 UTC (9,032 KB)
来源：arXiv：cs.LG · arxiv.org